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Executive Summary 


This is the final technical report of the Systems Engineering Research Center (SERC) research task RT- 
170. This research task (RT) addresses research needs extending prior efforts under RT-48/118/141/157 
that informed us that Model-Centric Engineering 1 (MCE) is in use and adoption seems to be 
accelerating. Model-centric engineering can be characterized as an overarching digital engineering 
approach that integrates different model types with simulations, surrogates, systems and components 
at different levels of abstraction and fidelity across disciplines throughout the lifecycle. Industry is 
trending towards more integration of computational capabilities, models, software, hardware, 
platforms, and humans-in-the-loop. The integrated perspectives provide cross-domain views for rapid 
system level analysis allowing engineers from various disciplines using dynamic models and surrogates 
to support continuous and often virtual verification and validation for tradespace decisions in the face 
of changing mission needs. 

NAVAIR senior leadership confirmed in late 2015 that NAVAIR must move quickly to keep pace with the 
other organizations that have adopted MCE and who continue to evolve at an accelerating pace 
enabled by the advances in computational and modeling technologies, and improved methods. In 
March of 2016, there was a Change of Command at AIR 4.0 (Research and Engineering) and NAVAIR 
leadership decided to accelerate the Systems Engineering Transformation (SET). During 2017 there was 
strategic planning to characterize the Functional Areas of the SET Execution Framework. One of those 
functional areas is the SET Research that is discussed in this report. This research provides analyses into 
NAVAIR enterprise capability, and builds on efforts for cross-domain model integration, model integrity, 
ontologies, semantic web technologies, multi-physics modeling, and model visualization that extend RT- 
157 research under RT-170 to address the evolving SET needs and priorities of SET. 

A key decision by NAVAIR leadership in 2017 was to conduct a surrogate pilot as reflected in Figure 1. 
The aspects of this research task are blended into the surrogate pilot. The surrogate pilot is using 
experiments to simulate the execution of the new SET Framework, shown in Figure 2. The first phase of 
the SET Surrogate Pilot addresses the following mission, goals and objectives: 

■ Mission: Collaboration between Government and Industry in Model-based Acquisition under 

SET Framework 

■ Goal: Execute SET Framework to Assess, Refine, and Understand a New Paradigm for 

Collaboration in Authoritative Source of Truth (AST) 

■ Objectives (non-exhaustive): 

o Formalize experiments to answer questions about executing SET framework using Surrogate 
Contractor (SC) 

o "Government team" creates mission, system (& other) models, "generates 

specification/Request For Proposal," and provides acquisition models to SC as Government 
Furnished Information (GFI) 

o SC refines the GFI to make corrections and add innovations with physical allocation views 
with an multi-physics-based Initial Balanced Design 


1 DASD has increased the emphasis on using the term Digital Engineering (DE). A draft definition provided by the 
Defense Acquisition University (DAU) for DE is: An integrated digital approach that uses authoritative sources of 
systems' data and models as a continuum across disciplines to support lifecycle activities from concept through 
disposal. This definition is similar to working definition used throughout our prior research task (RT) 48/118/141 
for Model Centric Engineering (MCE). 
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o Simulate continuous virtual reviews and derive new objective measures for assessing 
maturing design in AST 

o Demonstrate visualizations for real-time collaboration in AST 
o Demonstrate and document methods applied 
o Investigate challenging areas and research topics in series of pilots 



Figure 1. SE Transformation "Roll out" Strategy 
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While modeling everything may not be practical for all projects, the plan is to use models to the extent 
possible in order to demonstrate the feasibility and desired methods that will be captured as examples 
in reference models. The pilot is developing an experimental Unmanned Aerial Vehicle (UAV) system 
called Skyzer. The focus is on learning about a new operational paradigm between government and 
industry in the execution the SET Framework, not necessarily to produce an entire air vehicle design. 
There are many more detailed facets to the surrogate pilot that are discussed in this report, and the 
surrogate pilot, which had an official kickoff in December of 2017, is ongoing through 2018. 

The strategic plans of SET and overarching goals of this research have been expanded through RT-170. 
RT-170 has research collaborators from Stevens Institute of Technology, Georgia Institute of Technology 
and University of Maryland, in addition to the surrogate pilot team that includes a Surrogate 
Contractor, and team members from NAVAIR and NAVAIR contractors. We are also working 
collaboratively with US Army RDECOM-ARDEC in Picatinny, NJ under RT-168, and some of the research 
results derived from those efforts that are being leveraged in the surrogate pilot are discussed in this 
report. We are also leveraging research efforts from RT-176, the Naval Postgraduate School 
Collaborators. Additional research associated with this research is planned for RT-195. 
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1 Introduction 


In 2013, the Naval Air Systems Command (NAVAIR) at the Naval Air Station, Patuxent River, Maryland 
initiated research into a Vision held by NAVAIR's leadership to assess the technical feasibility of a radical 
transformation through a more holistic model-centric system engineering (MCSE) approach. The 
expected capability of such an approach would enable mission-based analysis and engineering that 
reduces the typical time by at least 25 percent from what is achieved today for large-scale air vehicle 
systems using a traditional document-centric approach. The research need included the evaluation of 
emerging system design through computer (i.e., digital) models. 

Through Systems Engineering Research Center (SERC) research tasks (RT-48, 118, 141, 157) there was 
considerable emphasis on understanding the state-of-the-art through discussions with industry, 
government and academia [23] [35]. The team comprised of both NAVAIR and SERC researchers 
conducted over 30 discussions, including 21 on site, as well as several follow-up discussions on some of 
the identified challenge areas and approaches for a new operational paradigm between government 
and industry. 

In 2015, the NAVAIR leadership concluded that they must move quickly to keep pace with the other 
organizations that have adopted MCE as the pace of evolution is accelerating by the enabling 
technologies. NAVAIR made the decision to press forward with a Systems Engineering Transformation 
(SET). That effort was started in January of 2016 under RT-157 and had four tasks as shown in Figure 3: 

■ Task 1 - Model Cross-Domain Integration with underlying Single Source of Technical Truth (SST) 

■ Task 2 - Model Integrity - developing and accessing trust in model and simulation predictions 

■ Task 3 - Modeling Methodologies aligning with the roll out of technologies defined under Task 4 

■ Task 4 - Define System Engineering Transformation Roadmap 


1) Model Cross-Domain Integration 


2) Model Integrity 


Targeted discussions with Government, Industry & 
Academia on developing and operating in modeling 
framework enabling 
cross-domain 
model integration 
& Single Source 
of Technical 
Truth (SSTT) 

methodology qg 



Define Methodologies for Model Integrity and 
Uncertainty Quantification: 

• Provide trust in model-based predictions, with 
Quantification of Margins & Uncertainties 

• Framework for integrating risk and understanding 
uncertainty in the data 



CM 1 rJZk 


DM CM 


1 - \ \ - X 


Model-Centric Methodology 



Develop a roadmap to rollout capabilities addressing 
all five perspectives in parallel: 

1. Technologies and infrastructure for SSTT 

2. Methodologies and processes 

3. People, competencies t 
and SSTT interfaces 

4. Operational & contractual ^ 

paradigms for transformed j — * —— 

interactions with industry 3|:r_ * 

5. Governance “_ 


3) Modeling Methodology 
Implementation at NAVAIR 


4) SE Transformation Roadmap 


Figure 3. SE Transformation Phase II 


In March of 2016, there was a Change of Command at AIR 4.0 (Research and Engineering). NAVAIR 
decided to accelerate the Systems Engineering Transformation (SET). Notionally as shown in Figure 4, 
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the research has a layered approach providing analyses into NAVAIR enterprise capability (e.g., 
modeling capabilities, reference models, and an authoritative source of truth), but builds on efforts for 
cross-domain model integration and model integrity (per RT-157). While the initial SERC research RT- 
48/118/141 was directed to focus on the Program of Record (POR)/systems level, a new NAVAIR 
strategy for accelerating capability delivery to the warfighter is looking to better assess the value and 
risks of system and system of systems (SoS) capabilities, potentially distributed across platforms to 
mission and campaign needs in a more dynamically changing environment. Therefore, NAVAIR added 
the following areas to the SERC research as characterized in RT-170, which are a layer on top of the 
other dimensions of the research, as shown in Figure 4: 


■ Prioritization and Trade-off Analysis 

■ Concept Engineering 

■ Architecture & Design Analysis 

■ Design & Test Reuse and Synthesis 

■ Active System Characterization 

■ Human-System Integration 


Modeling Methodology 
at NAVAIR 


SET Roadmap 




Figure 4. SE Transformation Research Areas (SERC) 


During the execution of RT-157, our NAVAIR sponsor, Dave Cohen, proposed a new operational 
framework which has evolved into a concept depicted by Figure 2. The research efforts during 2017 for 
RT-170 started developing a surrogate pilot concept to assess and refine the Execution of SET 
Framework through a series of experiments. The emphasis is on a new operational paradigm to mission 
engineering, analysis and model-based acquisition, which would be led by NAVAIR with a collaborative 
design efforts led by industry. We continue to participate with our sponsors in industry meetings to 
assist in communicating and clarifying and aligning these concepts for a new type of collaboration, and 
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to assess the impacts on the NAVAIR enterprise, from both a technical and socio-technical perspective. 
Many objectives for assessment and refinement are characterized as objectives and captured as part of 
a Surrogate Pilot Project plan and model that will be traced to experimental models, demonstrations 
and results. 

Briefly the concept of the new SET framework for transforming from a document-centric process with 
monolithic reviews to an event-driven model-centric approach involves, but is not limited to: 

■ A concept for collaborative involvement between Government and Industry to assess mission 
and System of Systems (SoS) capability analyses, where NAVAIR has the lead to: 

o Involve industry in SoS capabilities assessments during mission-level analysis (to the degree 
possible) 

o Iteratively perform tradespace analyses of the mission capabilities using approaches such as 
Multidisciplinary Design, Analysis and Optimization (MDAO) as a means to develop and 
verify a model-based specification 

o Synthesize an engineering concept system model characterized as a model-centric 
specification and associated contractual mechanism based on models or associated 
formalism 

■ At the contractual boundaries, industry will lead a process to satisfy the conceptual model 
addressing the Key Performance Parameters (KPPs), with particular focus on Performance, 
Availability, Affordability, and Airworthiness to create an Initial Balanced Design 

o Industry applies MDAO at the system and subsystem level too 

o There is a potential need to iterate back to re-balance the needs if the tradespace analyses 
of the solution/system for the program of record (POR) cannot achieve mission-level 
objectives 

o All requirements are tradeable if they do not add value to the mission-level KPPs 
o These are asynchronous activities in creating an Initial Balanced Design 
o Government and Industry must work together to assess "digital evidence" and "production 
feasibility" 

Another objective under consideration in the context of the operational model is to replace large-scale 
document-centric reviews such as Systems Requirements Review (SRR), System Functional Review 
(SFR), Preliminary Design Review (PDR), etc. with continual event-driven reviews using objective 
evaluation based on model-centric information. NAVAIR needs some type of objective decision 
framework to assess evolving design maturity with considerations of value to the KPPs, risk and 
uncertainty. This is another objective for the surrogate pilot. 

These efforts for improving the collaboration between government and industry are also underway to 
develop a new Concept of Operations (CONOPs) with organizations involved in the Aerospace Industry 
Association (AIA) working group [3], and the National Defense Industry Association (NDIA) Modeling 
and Simulation group which is looking at approaches for using digital engineering for competitive down 
select. We are involved in these efforts to further the objectives of our sponsor to communicate the SET 
Framework concept to industry and other Department of Defense stakeholders. 

This report covers research performed against the RT-157 objectives, and at the request of our sponsor 
has expanded to address the needs of the SET under RT-170 using experiments in the Surrogate Pilot 
Project using a hypothetical system called Skyzer. Skyzer has a CONOPs for an UAV that provides 
humanitarian maritime support use cases as reflected in Figure 5. Another objective that must be 
factored into the use cases must allow the surrogate contractor to demonstrate measures to support a 
production-readiness decision for a multi-physics design. 
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Figure 5. Graphical CONOPS for Skyzer UAV 


1.1 Objectives 

The objectives of the research are cross-cutting and interrelated, as shown in Figure 6. The research 
needs expansion on the prior research and include specific focus on technological aspects to address 
the research gaps in the context of the SET Framework, but still include cross-domain model 
integration, model integrity, ontologies, semantic web technologies, modeling methods, multi-physics 
modeling, and model visualization. We summarize and organize the research in a manner used on RT- 
168 as use cases (UC) that cut across the evolving case studies as it relates to Figure 6. 
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Figure 6. Cross-cutting Relationships of Research Needs 


The use cases include, but are not limited to: 

■ UCOO: Ontologies and semantic web technologies for reasoning about completeness and 
consistency across cross-domain model to achieve the notion of model integration through 
interoperability, which are enablers for an authoritative source of truth, tool-agnostic 
approaches to methodology enforcement and conformance that also support model integrity 

■ UC01: Multidisciplinary Design, Analysis and Optimization (MDAO) at the mission, system and 
subsystem levels, which provides a means for continual assessment of trades (i.e., analysis of 
alternatives) to support KPP assessment; this also relates to representations within system 
models 

■ UC02: Integrated Modeling Environment (IME) in the context of the workflows, which has 
implications on both technologies and workforce development 

o We are using an instantiation of OpenMBEE [118] as the experimental integrated modeling 
environment formalization of the AST, in the context of NAVAIR, but also in the context of 
one or more industry contractors 

o Model visualization from multiple perspectives including, but not limited to enabling 
different views relevant to different stakeholder (or due to particular access), reducing 
complexity, and analytical analysis 

o Methods for model modularization to ensure separation of concerns, classification, and 
acquisition 

o Methods for creating and organizing Enterprise, Process, and Reference models 
o Understanding the operational paradigm between industry and government in the context 
of the SET Framework through MCE 
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o Workflow analysis and representation relative to a program instantiation of tool suites from 
the IME 

o NOTE: cybersecurity and classification is not currently in the scope of this work 

■ UC03: Methodologies relevant to technologies in the context of the IME workflows, such as: 
o Methods for system model 

o Methods for mission model 
o Methods for MDAO modeling 

o Methods for modularizing models to support constraints needed for developing an 
Authoritative Source of Truth (AST), which relates to many other use cases 
o Methods for model management 

o Methods for representing and organizing reference models, process models, discipline- 
specific models 

o Methods for developing and tracing capabilities measure to KPPs 

o Alternative approaches to improve modeling methods, which is fundamental to ensuring 
model integrity (strong relationships to UC02) 

■ UC04: Model-physics modeling, which is also supported by MDAO and approaches for assessing 
model integrity risks and uncertainty 

■ UC05: Representation to formalize research under RT-176 in models to support requirement 
verification and validation using Monterey Phoenix 

■ UC06: Experimentation and learning about defined research topics in the execution of the SET 
through unclassified pilot programs, such as the SET Surrogate Pilot; this includes alignment 
with the SET Tasking and other research use cases with evolving pilot case studies (as described 
below) 

■ UC07: Research into Enterprise Transformation to support governance and workforce 
development (not covered in this report; initial funding is included under RT-195) 


1.2 Scope 

The scope for the research aligns the objectives as characterized by the use cases in Section 1.1 to the 
prioritizes of the SET in the context of the surrogate pilot for experimenting with the research to 
produce models and demonstrations for NAVAIR-relevant examples that can help inform the workforce 
and other stakeholders. These objectives include, but are not limited to understanding the methods, 
models, tools and processes to execute the SET Framework. 

The two perspectives that are further described throughout this report can be thought of as 
overarching use cases, as reflected by Figure 7: 

1. Use cases about the objectives for the Skyzer experiments and associated environments: 

o Surrogate Pilot Use Cases characterize objectives for understanding the execution of the SET 

Framework 

• A non-exhaustive set of objectives for the surrogate pilot are characterized in the SET 
Surrogate Pilot Project Model; an automatically generated version of the model content 
(e.g., "document") from this evolving model is provided in Section 3 

o Collaboration in an Authoritative Source of Truth (AST) Use Cases 

• The government side of the AST is being developed using the NASA/JPL OpenMBEE 
[118] and commercial modeling tools that is hosted on Amazon Web Services server 

• The surrogate contractor side of the AST must be "integrated" with the government side 
of the AST 
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2. Use cases for the Skyzer Experimental System using AST, which involves the development of 
evolving models for: 
o Surrogate Project/Planning Model 

• Characterizes the objectives for the surrogate pilot and research 

• Discussed in more detail in this report in Section 3 
o Project Planning for Skyzer 

• Currently this is a more traditional document 
o Mission Model for Skyzer 

• Parts of mission model provided as Government Furnished Information (GFI) 

• Primarily associated with Element 1 of SET Framework 
o System Model for Skyzer 

• Parts of system model provided as GFI 

• Primarily associated with Element 2 of the SET Framework 
o Acquisition Model Skyzer 

• Primarily associated with boundary between Element 2 and Element 3 of the SET 
Framework 

• Provide criteria for source selection evaluation 
o Surrogate Contractor System model for Skyzer 

• Surrogate contractor to assess, refine and extend GFI system model 

• Primarily associated with Element 3 of the SET Framework 
o Surrogate Contractor Design models for Skyzer 

• Design models much address aspects of multi-physics 

• Primarily associated with Element 3 and Element 4 of the SET Framework 
o View and Viewpoints for DocGen and other Libraries 




Contractor(s) 
System 
^Model(s) UC 


Contractor 
Design 
Model UC 


System 
Model UC 


Project Plan 
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Section UC 
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Mission 
Model UC 


Objectives to Assess SE Framework 
X 


How we Collaborate in AST 
X 


Figure 7. Use Cases for Surrogate Pilot and Experimental System (Skyzer) 


As shown in Figure 8, this concept involves developing operational models and user capabilities, which 
are primarily defined in the Skyzer Mission Model. The mission model(s) provides inputs that are 
captured in an "Initial System Model" that characterizes the "requirements" derived from models that 
will be part of a Request for Proposal (RFP) that would be elaborated by contractors during source 
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selection into a "Final System Model." This model is the Skyzer System Model and being developed by 
our Georgia Tech collaborator. We are simulating this concept during the pilots. Notionally, Figure 8 
shows the related alignment to the four Elements 1, 2, 3, & 4 with the focus being on formalizing SET 
using models. 



Figure 8. Characterizes the Boundary of Models between Government and Industry 

The research approach will use experimentation in evolving pilot project scenarios to help inform the 
workforce, as well as create reference models as examples to exemplify best practice methods. Our 
research is already investigating concepts that have never been attempted by NAVAIR. There are many 
questions that have surfaced related to the execution of the SET Framework, and we cannot address all 
of these during the surrogate pilot, but reflect on some of the broad needs, for example: 

■ Simulating Mission Engineering and Analysis (Element 1) and System modeling (Element 2) prior 
to contract award 

■ Formalization/synthesizing a "specification" from models for "RFP" and methods for providing 
models to contractor 

■ Simulating "Execution" of Oversight / Insight in a Single Source of Truth per SET Framework and 
capturing abstractions of "recommended" processes in potentially heterogeneous environments 
(Elements 3 & 4) 

■ Developing and assessing the use of objective measures for evaluating evolving design maturity, 
while assessing the reduction of risk and uncertainty 

■ Simulating feedback back to mission engineering caused by specified objectives for unachievable 
KPP 

■ Simulating approach for "faults in specification/model" detected after contract award to look at 
the potential needs for a new paradigm referred to as model-based acquisition 

■ Simulating source selection and investigate if it is possible to use dynamic simulations and V&V 
as part of the source selection process and evaluation criteria 
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■ Working with contracts/legal to get agreement on what a "specification" would or can be, while 
helping to understand potential needs to change acquisition policy 

■ Methods for modularizing model used to "generate specification" and for sharing digital models 
while addressing access needs such as security 

■ Assessing how or if we can use an ontological representation of the Systems Engineering 
Technical Review (SETR) guide and checklist that NAVAIR uses? And, how will we make 
recommendations for its evolution in the context of MCE 

■ Use of Multidisciplinary Design, Analysis and Optimization 


1.3 Collaborative Research Synergies 

Finally, NAVAIR is also involved in synergistic collaborative efforts with ARDEC and the Digital 
Engineering (DE) Working Group led by ODASD(SE), and we are working to align the research, to the 
extent possible, with the five DE Transformation goals that include: 

■ Gl. Formalize the development, integration and use of models to inform enterprise and 
program decision making. 

■ G2. Provide an enduring authoritative source of truth. 

■ G3. Incorporate technological innovation to link digital models of the actual system with the 
physical system in the real world. 

■ G4. Establish a supporting infrastructure and environment to perform activities, collaborate and 
communicate across stakeholders. 

■ G5. Transform a culture and workforce that adopts and supports Digital Engineering across the 
lifecycle. 

As is reflected in Figure 9, many of the research topics under investigation by the SET align with the DE 
Transformation goals. In addition, the mapping in Figure 9 shows that the research areas have 
significant overlap with some of the DE goals. This means that in order to achieve some of the goals, it 
will be necessary to have successful research outcomes across many research areas. Therefore, in the 
Part II of this report, the research areas are defined as cross-cutting use cases rather than specific tasks, 
which is similar to what evolved out of the RT-168 research task with ARDEC. 
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Future Research Areas 
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activities are best performed by human decision makers 
and what can effectively be automated or augmented with 
model intelligence 
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to work in a DE environment 
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MCE acquisition to understand the needed changes to 
acquisition and security when developing n the new DE 
environment 

X 
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Figure 9. Future Research Areas Mapped to Goals of Digital Engineering Transformation Strategy 


These use cases will investigate continuing synergistic research with the US Army ARDEC under RT-168, 
Semantic Technologies for Systems Engineering (ST4SE) initiative, RT-176 and other potential SERC 
research that is aligned with the principles and concepts for the SET as well as the ODASD(SE) Digital 
Engineering Strategy. 


1.4 Organization of Document 

Section 1 provides an overview of the context for the needed research, objectives, expanded scope and 
organization of this report. 

Section 2 provides a summary of our efforts, meetings, and deliverables. 

Section 3 includes a unique approach to the creation of Section 3, which is automatically generated 
from SET Framework Surrogate Project model. We include this section to illustrates the capabilities to 
leverage models and with the OpenMBEE Model Development Kit (MDK) DocGen capabilities. 

Section 4 describes the use case that investigates the development and use of ontologies and more 
generally semantic web technologies (SWT) for reasoning about completeness and consistency across 
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cross-domain models. These capabilities support enforcement of modeling methods and support for 
model integration through interoperability. 

Section 5 describes the use case that investigates various uses of Multidisciplinary Design, Analysis and 
Optimization (MDAO) at the mission, system and subsystem levels, which provides a means for 
continual assessment of trades (i.e., analysis of alternatives). This section also discusses methods for 
applying MDAO, and a modeling approach that allows SysML models to be transformed into MDAO 
workflows. 

Section 6 describes the use case that investigates topics for Integrated Modeling Environments (IMEs) 
with specific focus on creating and collaborating in an AST for the surrogate pilot in the context of the 
research thrusts, and the specific instantiation of the IME that is being used for the surrogate pilot. 

Section 7 discusses the use case that investigates the development and demonstrations of methods for 
mission, systems, model modularization and management for technologies in the context of the IME 
workflows. 

Section 8 discusses the use case that investigates model-physics modeling, MDAO and model integrity 
which is also supported by MDAO and approaches for assessing model integrity risks and uncertainty. 

Section 9 discusses the use case that investigates the development of SysML representations to 
formalize the Monterey Phoenix research under RT-176 support requirement verification and 
validation. 

Section 10 discusses the use case that defines the objectives for experimentation and learning in the 
execution of the SET, which is related to surrogate pilot plan and model described in Section 3. 

Section 11 is a placeholder for a task that is to be funded under RT-195 for Enterprise Transformation to 
support governance and workforce development. 

Section 12 summarizes some of the SERC research synergies. 

Section 13 provides a summary and some conclusions. 
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2 Research Events and Deliverables Summary 


This section provides a high-level summary of the research-related events, results and deliverables. 
Unlike prior final reports, we are not including a historical perspective of prior research, but have 
shifted focus to the recent developments in 2017 for the SET, with specific focus on the research 
addressed through the surrogate pilot. The technical reports RT-141 [23] and RT-157 [29] provide a 
comprehensive summary and historical perspectives leading up to the SET. 

Part II of this report includes a summary of the Surrogate Pilot Project that is the venue in which we 
work on the cross-cutting research. Section 3 has been automatically generated from the SET 
Framework Surrogate Project model and provides significant details on the surrogate pilot. In addition, 
Part II provides details on each use case, as well as the research synergies leveraged from the ARDEC 
research under RT-168 [26]. 


2.1 Working Sessions and Sponsor-Supporting Events 

A component of the research and required deliverables are conducting working sessions that inform the 
NAVAIR team about progress against the plan. These working sessions also inform the team about 
relevant information and feedback to scope the deliverables in the context appropriate for NAVAIR; this 
has been especially important given the recent changes under SET. We also use bi-weekly drumbeats to 
share status and updates. Each working session has a defined agenda, and detailed meeting notes are 
provided to our sponsor. The following provides a summary of the working sessions and other events, 
and a brief description of the contributions to the tasks and deliverables. 

■ Participation in Bi-weekly Drumbeat 

■ Working session at NAVAIR February 2, 2017 

■ Working session at NAVAIR March 9, 2017 

■ Special session to discuss SET at Northrop Grumman, El Segundo, CA with Dave Cohen, Jaime 
Guerrero and David Meiser, March 14, 2017 

■ Special session to discuss SET at NASA/JPL, Pasadena, CA with Dave Cohen, Jaime Guerrero and 
David Meiser, April 20, 2017 

■ Special session to discuss SET at Lockheed Martin, Ft. Worth, TX with Dave Cohen, Jaime 
Guerrero and David Meiser, April 20, 2017 

■ Working session at NAVAIR May 4, 2017 

■ Working session at NAVAIR June 8, 2017 

■ Working session and pre-meeting on surrogate pilot at NAVAIR July 12 & 13, 2017 

■ Working session and follow-up meeting at NAVAIR, August 16 & 17, 2017 

■ Working session and follow-up meeting at NAVAIR, September 19 & 20, 2017 

o Official Kickoff of the NAVAIR SE Transformation by Dave Cohen on September 19 

■ Special Briefing of the SET with Kristin Baldwin, Jim Thompson, Phil Zimmerman, Tracee Gilbert, 
and Jared Kaib at NAVAIR, October 2, 2017 with Dave Cohen, Jaime Guerrero and David Meiser 

■ Working session at NAVAIR October 19, 2017 

■ Working session at NAVAIR November 16, 2017 

■ Surrogate Pilot Kickoff at NAVAIR December 7, 2017 with Dave Cohen, Jaime Guerrero and 
David Meiser 

■ Conducting weekly meetings with Surrogate Contractor and Surrogate Pilot Team, which started 
December 2017; updates on meeting in All Partners Access Network (APAN) 

■ Working session at NAVAIR January 25, 2018 
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Other related NAVAIR/SERC events: 

■ Attended NASA/JPL Symposium and Workshop on Model Based Systems Engineering, January 
25-27; there is a separate report that covers the entire event 

■ Attended INCOSE International Workshop, January 28-31, 2017; there is a separate report that 
covers this event 

■ Helped with pilot MAGTF Agile Networking Gateway Link (MANGL) Overview for MBSE; 
collaborate with John Funk 

■ MBSE/SysML workforce training supported by Georgia Tech 

■ Semantic Technology for Systems Engineering (ST4SE); Dinesh Verma initiated an effort with the 
support of Chi Lin, Steve Jenkins and Mark Blackburn to bring a community of people together in 
an attempt to create and ecosystem on Semantic Web Technologies 

o Started with a meeting was held at NASA JPL on March 22 nd on the subject 
o Core ST4SE team general meets bi-weekly and there have been three face-to-face meetings 

■ Bi-weekly participation in the Open Collaboration Group for MBSE that is providing support for 
adopting and contributing to OpenMBEE 

o This was critical to our success in deploying OpenMBEE for the Surrogate Pilot 

■ We are participating in the Aerospace Industry Association (AIA) CONOPS for MBSE 
Collaboration 

o This is a follow-up to the effort completed last year which developed a white paper on the 
Life Cycle Benefits of Collaborative MBSE Use for Early Requirements Development [3] 
o We have informed many industry providers to NAVAIR about the Surrogate Pilot 

■ Meeting NASA/JPL with Dave Cohen, Jaime Guerrero, David Meiser and Mark Blackburn to 
discuss concept of SET with NASA/JPL team and understand NASA's use of MCE on projects such 
as Europa and Asteroid Rendezvous & Retrieval Mission projects 

■ Industry meetings with Dave Cohen, Jaime Guerrero, David Meiser and Mark Blackburn to 
discuss new operational paradigm for executing against SET between industry and government 

■ Involved in collaboration with RT-176, called Verification and Validation (V&V) of System 
Behavior Specifications. Jaime Guerrero wanted to ensure that RT-176 aligns with the objectives 
from RT-157 and RT-170 and asked Mark Blackburn to be involved in RT-176 as a Co-PI 

■ Special Session: 31-July-2017 held at Stevens Institute of Technology 

o This special session invited our sponsors from ARDEC, NAVAR, and DASD(SE), but also other 
organization Naval Surface Warfare Center, Digital Warfare Office, and MITRE, and industry 
guests from Raytheon working on Semantic Web Technologies and Ontologies 
o Objectives included: "Provide Big Picture - Mental Model" 
o "Past - Why" - Historical perspectives - How we got here and why 

o "Present - What" - Aligning the research gaps and challenges for a Systems Engineering 
Transformation 

o "Future - How" - Blending and evolving our research results with Digital Engineering (DE) 
Transformations across the Department of Defense (DoD) to be in a Future State by 
Computationally Enabled DE 
o Deep Dive a Few Research Topics 

o Integrated Systems Engineering Decision Management (ISEDM) Process Enabled by Digital 
Engineering Technologies, presented by Dr. Matthew Cilli 
o Semantic Technologies and Ontologies Research to enable Trade Space Analytics for 
Engineered Resilient Systems, presented by Dr. George Ball 
o Breakout Session discussing 

• Risk for Digital Engineering Transformation 
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• Priorities for Digital Engineering Transformation 

■ National Defense Industry Association (NDIA) - presented Surrogate Pilot Overview, October 25, 
2017 

■ SERC Sponsor Research Review (SSRR) - presented Surrogate Pilot Overview, November 8, 2017 

■ Ontology Boot Camp sponsored by SERC on December 5, 2017 


2.2 Deliverables 

As required by the contract, we produced: 

■ Technical Management and Work Plan 

■ Interim Technical Report 

■ Bi-monthly status reports 

■ Meeting Notes and Briefings 

■ Final Technical Report 

We have also produced models, demonstrations, videos, examples and assembled tools for an 
Integrated Modeling Environment (IME) for the surrogate pilot. The following provides a list of models 
that have been produced and supplied to NAVAIR: 

■ APAN (apan.org) 

o Tracking progress of the surrogate pilot using Discussion group that is linked to related 
evolving artifacts 

o Posting documents into both the general NAVAIR SET area as well as the Research area 

■ Successfully created instantiation of OpenMBEE both at Stevens and on Amazon Web Services 
(AWS) to be used in the surrogate pilot 

■ Demonstration of OpenMBEE Model Development Kit (MDK)/DocGen 

■ Surrogate Project Plan Model 

■ Surrogate Mission Model for Skyzer 

■ Surrogate System Model for Skyzer 

■ Skyzer Request for Information package 

■ Briefing on creating SysML Activity Diagram for Monterey Phoenix in support of RT-176 

■ MDAO demonstrations 

■ Videos for the operations of OpenMBEE with Teamwork Cloud to be used on surrogate pilot 

■ SysML one-day course that is provided by Stevens Institute of Technology for the SYS- 
671,672,673,674 Cyber Physical Systems course 
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Part II: Task Detail Summary 

Part II includes a unique approach to the creation of Section 3, which has been automatically generated 
from SET Framework Surrogate Project model, and then inserted into this report. We include this 
section to illustrates the capabilities to leverage models with the OpenMBEE Model Development Kit 
(MDK) DocGen capabilities. The Document Generator (DocGen) is a module of MDK plug-in in 
MagicDraw. It provides the capability to generate formal documents from UML/SysML models in 
MagicDraw. A "document" is a view into a model, or a representation of model data, which may be 
structured in a hierarchical way. We are able to produce information extracted from that model using 
View and Viewpoints [118]. It is included here to provide examples of the types of research we are 
conducting and to provide an example to illustrate that it is possible to completely develop 
"documentation" and "specifications" using models and associated modeling technologies like DocGen. 
Note also the figure and table number comes directly from the generation process. The latest updates 
will be available from the Amazon Web Service which is hosting OpenMBEE and the surrogate pilot 
models or on APAN [7]. The only modifications that we made to the generated content in Section 3 is to 
use the standard SERC Heading styles in this report and to resize the figures. 

The remainder of the sections provides additional details associated with the research use cases listed 
in Section 1.1. An extensive amount of material covered in Part II of the RT-141 final report [23] and RT- 
157 final report [29] is still relevant information to this research, but has not been included in this 
report. For the convenience of the readers, we include some of the key topics from those reports: 

■ Traceability and scope of data collection of state-of-the-art MCE relevant topics collected from 
global scan of industry, government and academic 

■ Characterization of canonical reference architecture of an Integrated MCE Environment 

■ Model cross-domain integration within the underlying single source of truth 
o Information Model for a Single Source of Technical Truth 

o Requirement ontology 

o This topic is still relevant and discussed in this report 

■ Model Integrity - developing and accessing trust in model and simulation predictions 

■ Modeling methodologies 

■ Multidisciplinary Design, Analysis and Optimization (MDAO) 

■ Example models 

■ Modeling and Methods for Uncertainty Quantification 

o Dakota Sensitivity Analysis and Uncertainty Quantification (UQ) 
o Overview of Quantification of Margins and Uncertainty 

■ Modeling Methods for Risk 

■ Controlled Natural Language Requirements information 

■ Cross-Domain Integration and Natural Language Process of Requirements using Ontologies 

3 NAVAIR - SERC Systems Engineering Transformation Surrogate Pilot 


3.1 SE Transformation Surrogate Pilot Project 

NAVAIR Public Release 2017-892. Distribution Statement A - ’’Approved for public 
release; distribution is unlimited” 
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Certain commercial software products were used to create this material. These products were used only 
for demonstration purposes. This use does not imply approval or endorsement by Stevens, SERC, or 
ARDEC, NAVAIR, nor does it imply these products are necessarily the best available for the purpose. 
Other product names, company names, images, or names of platforms referenced herein may be 
trademarks or registered trademarks of their respective companies, and they are used for identification 
purposes only. 

Tue Jan 30 11:38:07 EST 2018 


3.2 Chapter 1. Surrogate Pilot Overview 
Table of Contents 

Use Cases 

Surrogate Project Modeling Approach 

This is a work in progress. This document was completely generated from a combination of models as 
described in this document. 

The purpose of this project is to simulate the Execution of the new Systems Engineering Transformation 
(SET) Framework using a "completely” model-centric approach. Therefore, while modeling everything 
may not be practical for all projects, the plan is to attempt to use models exclusively in order to 
demonstrate the feasibility and desired approaches that will be captured in reference models. The current 
model defines the first phase of the Surrogate Pilot. 

Mission: Collaboration between Government and Industry in Model-based Acquisition under SET 
Framework 

Goal: Execute SET Framework to Assess, Refine, and Understand a New Paradigm for Collaboration in 
Authoritative Source of Truth (AST) 

Objectives (non exhaustive - see Surrogate Pilot Objectives) : 

• Formalize experiment to answer questions about executing SET framework using Surrogate 
Contractor (SC) 

• ’’Government team” creates mission, system (& other) models, “generates specification/RFP,” & 
provides acquisition models to SC as Government Furnished Information (GFI) 

• SC refines GFI reflects corrections/innovations with physical allocation views with multi- 
physics-based Initial Balanced Design 

• Simulate continuous virtual reviews and derive new objective measures for assessing maturing 
design in AST 

• Demonstrate visualizations for real-time collaboration in AST 

• Demonstrate and document methods applied 

• Investigate challenging areas and research topics in series of pilots 

The main components of this model are shown from different views to include: 


20 







1. The Surrogate Project/Planning Model (this component) 

2. The Project Planning Model for Skyzer 

3. Surrogate Mission Model for Skyzer 

4. Surrogate System Model for Skyzer 

5. Surrogate Acquisition Model Skyzer 

6. View and Viewpoints for DocGen and other Libraries 

We focus on learning about a new operational paradigm between government and industry in the 
Execution the SET Framework (NOT an air vehicle design). There are many more detailed facets to the 
surrogate pilot. The following is a non-exhaustive list of examples that are formalized as mission 
objectives for the surrogate pilot using a model: 

• Simulating prior to contract award 

• Formalization of a “specification” for “RFP” and methods for providing models to contractor 

• Simulating “Execution” of Oversight / Insight in AST per SET Framework for real-time 
collaboration in heterogeneous environments 

• Objective measures for evaluating evolving design maturity, with the reduction of risk 

• Simulating feedback back to mission engineering caused by specified objectives for unachievable 
KPP 

• Simulating approach for “faults in specification/model” detected after contract award 

• Simulating source selection - desirably as a dynamic simulations and V&V 

• Working with contracts/legal to get agreement on what a “specification” would be 

• Methods for modularizing model used to “generate specification” 

• How will we use the SETR guide and checklist that NAVAIR uses? And, how will we make 
recommendations for its evolution 

• Applying research concepts such as: 

o Cross-domain model integration 
o Model integrity 

o Ontologies and semantic web technology 

o Use of Multidisciplinary Design, Analysis and Optimization (MDAO) 
o Modeling methods 

o Demonstrations bring this research together using OpenMBEE 

Figure 1.1. SET Framework 
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The new operational paradigm starts with mission engineering, analysis and acquisition led by 
government (Elements 1 & 2), and a collaborative design effort led by industry (Elements 3 & 4). Briefly 
the concept of the new SET framework for transforming from a document-centric process with monolithic 
reviews to an event-driven model-centric approach involves, but is not limited to: 

• A concept for collaborative involvement between Government and Industry to assess mission and 
System of Systems (SoS) capability analyses, where NAVAIR has the lead 

• Involve industry in SoS capabilities assessments during mission-level analysis (to the degree 
possible) 

• Iteratively perform tradespace analyses of the mission capabilities using approaches such as 
Multidisciplinary Design, Analysis and Optimization (MDAO) as a means to develop and verify a 
model-based specification 

• Synthesize an engineering concept system model characterized as a model-centric specification 
and associated contractual mechanism based on models or associated formalism 

• At the contractual boundaries, industry will lead a process to satisfy the conceptual model 
addressing the Key Performance Parameters (KPPs), with particular focus on Performance, 
Availability, Affordability, and Airworthiness to create an Initial Balanced Design 

• Industry too applies MDAO at the system and subsystem level 

• There is a potential need to iterate back to re-balance the needs if the tradespace analyses of the 
solution/system for the program of record (POR) cannot achieve mission-level objectives 

• All requirements are tradeable if they don’t add value to the mission-level KPPs 
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There are asynchronous activities in creating an Initial Balanced Design Government and 
Industry must work together to assess “digital evidence” and “production feasibility” 


3.2.1 Use Cases 


Figure 1.2. SET Surrogate Pilot Use Cases 



Notionally, these are the primary use cases that are further defined within this surrogate project plan using 
a method based on the NASA/JPL Integrated Model Centric Engineering (IMCE) ontologies. 


Table 1.1. Use Cases 


Model Element 

Documentation 

00 Refine SET 
Framework 

The main use cases is to Refine the SET Framework using a pilot project for 
simulating experiments while Executing the SET Framework. 

01 Identify 

Stakeholders 

Defined in following sections. 

02 Define Surrogate 
Project Plan 

This is what is reflected in this model. It is the plan about how to do Surrogate 
Experiments for assessing and refining the SET Framework. This document is 
produced from the model for the SET Surrogate Pilot. 

03 Define Project 
Plan 

This is the project plan for the surrogate system, currently referred to as Skyzer. 

04 Define Mission 
Model 

The Mission scenarios defined in Skyzer IM20. 

05 Define System 
Model 

The System model defined in Skyzer IM30. 
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Model Element 

Documentation 

06 Define Project 
Methods 

The project methods covering various processes. This is the Skyzer project plan. 
There is a need to identify a stakeholder that can operate as the lead of the project. 

07 Define 

Dependability Model 

This use case is for modeling dependability, which include reliability, safety, etc., 
and would use modeling techniques based on Hazard Analysis and Failure Modes 
and Effects Analysis. A stakeholder needs to be identified to support this use case. 

08 Define Logistics 
Model 

This is a logistics model. We need to identify stakeholders) that can support this 
use case. 

09 Simulate Source 
Selection 

The use case is about simulating the source selection. 

09.1 Define S ource 
Selection Evaluation 
Criteria 

This is the source selection evaluation criteria. This will likely evolve through 
additional pilot use cases. 

10 Collaborate in 
Authoritative Source 
of Truth 

This is the process by which we create and collaborate in the Authoritative Source 
of Truth (AST). This will involve the surrogate contractor(s) to ’’integrate" their 
environments with the NAVAIR surrogate pilot project IME. 

10.1 Define SET 
Integrated Model 

Environment 

This is the Integrated Model Environment (IME) that will be created by the 
NAVAIR surrogate pilot team, which is based on system modeling tools and 
OpenMBEE, the open source environment from NASA/JPL. 

11 Capture Lessons 
Learned 

This is a general set of lessons learned that we plan to capture from various internal 
and external stakeholder (e.g., other government organization and industry). This 
use case may include a more formal request for evaluating the surrogate pilot by 
directly examining the progress captured in the Authoritative Source of Truth 
(AST). 

12 Identify 

Additional Pilot Use 
Cases 

Define addition pilot use cases as we proceed through the execution of the first 
phase of use cases. Other examples include: mission systems, legacy system, 
Capability-Based Test and Evaluation, etc. 

20 Define Design 
Models 

This is the design created by the Surrogate Contractor. There will be many 
objectives placed on this use case that need to be assessed as part of a new 
operational paradigm between Government and Industry. 


3.2.2 Surrogate Project Modeling Approach 

There are several methods used to develop the NAVAIR surrogate pilot models (project, mission, 
system), and while there are a few traditional system model (SysML) views included in this model, this 
SET Framework Surrogate Pilot Project model uses the NASA/JPL IMCE ontologies as part of the 
Systems Engineering Research Center (SERC) research for this project. The figure shows a Partial Map 
of Foundation Ontology Concepts presented in a Module produced by NASA/JPL's Steve Jenkins. More 
information can be found here: https://nescacademy.nasa.gOv/category/3/sub/17 

Some example ontology definitions are included at the beginning of several section to illustrate how the 
ontology classes provide a basis for creating legal statements (as models) to characterize this project 
model. For example: 
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For clarification purposes these definitions were extracted from the NASA/JPL IMCE ontology. The 
ontologies have been transformed into profiles. Stereotypes from these profiles are used to allow the 
creation of legal sentences (axioms) about stakeholders, concerns, missions, objectives, projects, 
requirements, and components that comply with the ontologies. A few examples are provided here. 

A stakeholder is a person or organization with a recognized interest in the successful completion of a 
Project or Program. Example stakeholders include: executives, subject matter experts, engineers, 
and industry contractors. 

A Person corresponds to an individual named person. A Person belongsTo zero or more 

Organizations. 

A Role corresponds to a set of assignments meant to be filled by a single Person. 

A Mission is a Perf ormingElement that pursues Objectives. 

A Mission may contain Components, but the preferred relationship is that a Mission deploys its 
systems (which are Components). This convention allows for a Mission to be associated with shared or 
external Components that it does not strictly contain. 

An Objective represents a specific interest that one or more stakeholders have in the successful 
execution of a mission. Example Objectives include charactering how to Execute the SET Framework. 

Objectives differ from Requirements in that they are not the result of negotiated agreement between 
customer and supplier, they need not be mutually consistent, and a Mission pursues but need not 
completely achieve all its Objectives. In a sense, the set of Requirements for a Mission represents 
the minimum acceptable achievement of Objectives for a given cost, schedule, and risk. 

Figure 1.3. Surrogate Project Modeling Approach 
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pkg [Package] Surrogate Project Modeling Process [ Surrogate Pilot Ontology 


This is a Partial Map of Foundation Ontology Concepts presented in a Module 
produced by NASA/JPL's Steve Jenkins. 

As part of the research for the surrogate pilot, the method used to develop this 
surrogate pilot project model is based on using the NASA/JPL ontology and 
associated tools that are being released. We also expect to get some support from 
Steve Jenkins. 
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• "Requirement specifies Component" 

• "Component performs Function" 
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As part of the research for the surrogate pilot, the method used to develop this surrogate pilot project 
model is based on using the NASA/JPL ontology and associated tools that are being released. We also 
expect to get some support from NASA/JPL's Steve Jenkins. 


3.3 Chapter 2. Surrogate Pilot Context 
Table of Contents 

SET Surrogate Pilot Context 

Stakeholders 

External Stakeholders 

Internal Stakeholders 
Surrogate Contractor Stakeholds 
Concerns 

Assess and Refine SET Framework 

Modeling and Collaboration Environment Concerns 
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3.3.1 SET Surrogate Pilot Context 

As discussed in Chapter 1, this project plan is modeled to comply with the NASA/JPL IMCE ontologies. 
A Project is a kind of Authority that supplies a related set of Missions in pursuit of a set of 

Objectives. Stakeholders represent Objectives. 


Figure 2.1. SET Surrogate Pilot Context 


pkg [Package] SET Surrogate Pitot Context! SET Surrogate Pitot Context ] ) 


This is a partial representation of the SET Surrogate Pilot Context. 

Based on the NASA/JPL Ontology: 

A Project is a kind of Authority that supplies a related set of Missions in 
pursuit of a set of Objectives. 
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This is the Project Definition for 
the Surrogate System that is to 
be "designed" in the experiment. 


I'm really uncertain of these 
stereotypes and need to 
discussionwith Steve Jenkins. 
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Project or Program 
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Surrogate Evaluation 


3.3.2 Stakeholders 

This is an initial list of stakeholders. Specific individuals are identified when possible, otherwise 
organizations that need to provide stakeholders are identified. 

3.3.2.1 External Stakeholders 


Table 2.1. External Stakeholders 
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Model 

Element 

Documentation 

Government 

Government individuals that may participating in observing and commenting on the 
surrogate pilot. 

Industry 

Contractor 

Industry contractors that may participate in reviewing the surrogate pilot. 

NASA JPL 

This includes individuals that may contribute such as Chris Delp the lead on OpenMBEE 
and reviewers from NASA/JPL such as Chi Lin and Steve Jenkins. 

SERC 

Sponsor 


ST4SE 

This is the Semantic Technologies four Systems Engineering (ST4SE) team, which 
includes Mark Blackburn, Mary Bone, Steve Jenkins from NASA/JPL, Barry Smith, Chris 
Paredis, Henson Graves, etc. may be able to help apply ontologies to this program. 


3.3.2.2 Internal Stakeholders 

At the request of the NAVAIR sponsor, the list and roles of the internal stakeholders has been 
removed. 

3.3.2.3 Surrogate Contractor Stakeholders 

At the request of the NAVAIR sponsor, the list and roles of the surrogate contractor stakeholders 
has been removed. 


3.3.3 Concerns 

A Concern represents a specific interest that one or more stakeholders have in the successful 
completion of a Project or Program and its Missions. 


A Mission is a Perf ormingElement that pursues Objectives. 

An Objective represents a specific interest that one or more stakeholders have in the successful 
execution of a mission. Example Objectives include charactering how to Execute the SET Framework. 

For this reason, and because we are formalizing objectives in this model, many of the concerns have been 
formalized as more specific objectives that need to be characterized throughout this effort 

Objectives are elaborated in the following chapter. 


3.3.3.1 Assess and Refine SET Framework 

Table 2.4. Assess and Refine SET Framework 


Model Element 

Documentation 

Must right size the 
Capability Description 

Document 

Some examples characterized by Dave Cohen: 

1. Narrow top of the requirements pyramid 
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Model Element 

Documentation 


2. Off-load requirments to other elements of SoS and via TTPs 
(CONOPS) 

3. KPPs must be tied to mission effectiveness, Ao or Cost 

The Systems Engineering 
Technical Reviews events 
takes too long 

The traditional process for performing Systems Engineering Technical 
Review (SETR) events takes too long, happens too late, and does not take 
advantage of capabilities that would permit more continuous and 
asynchronous events. 

Time to develop 

capabilities is too long 

The time it takes to get new capabilities into the field is not keeping pace 
with the changing threats. 


3.3.3.2 Modeling and Collaboration Environment Concerns 

Table 2.5. Modeling and Collaboration Environment Concerns 


Model Element 

Documentation 

Ability to ensure Enterprise Governance to Modeling Environment 


Ability to share with stakeholders 


Ability to work and collaborate in an unclassified environment 


Ability to work and collaborate in classified environment 



3.4 Chapter 3. Surrogate Pilot Objectives 
Table of Contents 

Executing SET Framework Objectives 

Skyzer Phase I Objectives 

Skyzer Phase II Objectives 
Modeling Methods Objectives 
Real-time Collaborate in AST Objectives 

Decision Framework Objectives 

Source Selection Objectives 

Operations and Policy of Contracting Objectives 

Visualization Objectives 
Surrogate Evaluation Objectives 

This section starts from the Mission perspective of the SET Framework Execution. This section is 
evolving and presents a non-exhaustive set of mission objectives for Executing the SET Framework. 
These objectives will be defined and related, and there will be traceability created to show how the 
objectives are satisfied during the execution of the SET framework through the development of the 
Mission, System, and Design models. 
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The model representations used in this section are based on the NASA/JPL Mission ontology, which 
defines concepts for describing missions in terms of: 

• Objectives (this section) 

• Constituent components 

• Functions those components perform 

• Requirements that specify them 

The objectives are organized into about 10 classes that are presented in subsections of the chapter. Each 
subsection has one or more models (diagram) of the objectives that associate key stakeholder(s) with one 
or more objectives. These models formalize information and relationships that have been evolving in 
Power Point briefings. At the end of each section is a table that provides more information about each 
objective. 

Objectives differ from Requirements: 

• They are not the result of negotiated agreement between customer and supplier 

• They need not be mutually consistent 

• A Mission pursues but need not completely achieve all its Objectives 


3.5 Executing SET Framework Objectives 


Figure 3.1. Executing SET Framework 
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Figure 3.2. Executing SET Framework Authorities 
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bdd [Package] Executing SET Framework [ Executing SET Framework Authorities }\ 
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This view uses the «project:hasResponsibilityFor» relation to show those project stakeholders that 
have the primary responsibility for mission objectives. 


Table 3.1. Executing SET Framework Objectives 


Model Element 

Documentation 

Characterize Authoritative 
Source of Truth 

Develop a prototype infrastructure that can be used by both 
Government/SERC team to support Element 1 & 2, and that can also be 
"integrated" while conducting Insight/Oversight for collaboration with 
Surrogate contractors in Element 3 & 4. 

Characterize SET 

Framework Element 1 

Element 1 is fundamentally about mission modeling, but has other aspects. 
For example, does it include how the Integrated Warfare Analysis establishes 
CONEMPS and Effects-Chains and how they are modeled at the System of 
Systems (SoS) level? Therefore the objective is about characterizing all 
facets associated with Element 1. 

Characterize SET 

Framework Element 2 

Element 2 is also fundamentally about developing a System Model, 
synthesizing a specification, but it also factors in aligning the system model 
with the mission model and Key Performance Parameters. 

Characterize SET 

Framework Element 3 

Element 3 starts when the contract has been awarded. Element 3 is where the 
contractor refines the design awarded under contract. The key aspects for the 
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Model Element 

Documentation 


pilot is to understand how subject matter experts from NAVAIR are able to 
view, measure maturing designs in a collaborative environment (the AST). In 
addition, the pilot seeks to assess new operational approaches to contract 
modifications that are performed directly in models. 

Characterize SET 

Framework Element 4 

It is unclear what happens in Element 4. 

Characterize Source 

Selection in the context of 
the SET Framework 

The objective is to perform source selection in the context of models. The 
objective is to have the Government Furnish Information (models) that are 
provided as part of the RFP, be elaborated, corrected and refined by the 
surrogate contractors. We need to characterize exactly when this happens 
between Element 2 and Element 3 and all of the rules that govern Industry 
and Government collaboration. 

Formalize Experiment to 
Simulate Generating 

Specification From 

Mission and System 

Models 

This section of the model is characterizing many of the objectives that need 
to be formalized as experiments, for determining how mission and system 
models are used to generate a ’’specification" directly from models. The 
objective should potentially go beyond what might actually be needed in 
terms of modeling to demonstrate "how” rigorous and comprehensive 
modeling can be done. The experiments must also "seed" defects in the RFP 
delivered models to allow for understanding potential change management 
approaches in model-based acquisition under the SET Framework. 

Formalize SET Framework 
Process 

While it may not actually be necessary or possible to fully characterize the 
SET Framework Process, there are research merits to illustrate the concept of 
a process model, especially specific types of feedback loops that are related 
to operational interactions between the Government and the Surrogate 
contractor. 

Incorporate SERC 

Research 

The key research topics are: cross-domain model integration, model integrity, 
modeling methods, ontologies and many derived topics such as working 
collaboratively an authoritative source of truth (AST), which leads into the 
Integrated Modeling Environment (IME). A key reason for creating this type 
of model for the SET Surrogate Pilot project is to satisfy research 
requirements characterized in the SERC research task for the SERC 
collaborators. 
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3.6 Skyzer Phase I Objectives 


Figure 3.3. Skyzer Phase I 

bdd [Package] Skyzer Phase I [ Skyzer Phase I ]J 


•project: has Responsibility or* «project:hasResporsibilityFor» 



This model illustrates the «base:aggregates» relation from the IMCE ontologies, which provides a 
means to relate objectives. 


Table 3.2. Skyzer Phase I Objectives 


Model Element 

Documentation 

Characterize a CONOPS for 
Skyzer Phase I Mission 

This should define the CONOPS that will be refined by the Mission 
Model. 

Characterize a System model 
that aligns with mission model 

This is the system model that will be used with Bridging 
method/mechanism to produce both the generated specification as well as 
the Government Furnished Information (GFI) that will be part of the 
Request for Proposal (RFP) and used in source selection. 

Characterize Bridging method 
for deriving GFI models 

This is a concept for taking analysis derived modeling and bridging the 
specific information that is needed to go into a Government Furnished 
Information (GFI) model for purpose of Request for Information (RFI), 
Request for Proposal (RFP) and/or source selection. 

Characterize Mission 

modeling method that 

complies or refines Integrated 
Capability Framework 

The current approach for performing mission area analysis is based on 
the Integrated Capability Framework. This objective should demonstrate 
how model support or subsumes these guidelines. 
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Model Element 

Documentation 

Characterize Skyzer Project 
Model 

The objective is to define the Skyzer Project modeling guidelines, which 
currently includes the characterization of mission and system modeling 
methods. Does this occur in Element 1? 


3.7 Skyzer Phase II Objectives 


Figure 3.4. Skyzer Phase II 
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Scott is lead for the SET Framework Links. 


Table 3.3. Skyzer Phase II Objectives 


Model Element 

Documentation 

Characerize 

considerations for 

Operations and 

Sustainment 

Determine what type of information should be captured during the early 
stages of SET Element 1 and 2 phases that will better help with Operations 
and Sustainment. 

Characterize Capability- 
Based Test and Evaluation 

This is the concept discussed by Jim Carroll. The objective is to bring this 
capability in early and understand the implications on process. This may have 
impacts on source selection criteria. 

Characterize Legacy 

Systems scenarios 

This objective looks to evaluate and characterize how the SET Framework 
should be used for legacy systems. This particular question came from 
industry during briefings on the surrogate pilot. 

Characterize Mission 

Systems scenarios 

There is a belief that most ongoing changes and future changes will involve 
mission systems, and Phase II of the surrogate pilot should be structure like a 
block upgrade, which provides an opportunity to provide scenarios for 
involving Mission Systems for the flight vehicle. 

Characterize Model Based 
Test Engineering 

This should bring in the capabilities of the Model Based Test Engineering 
research performed by Jim Ciarcia. This includes metamodels that represent 
and ontology for many phases of this process. The addition objectives are to 
bring these concepts in early. 
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Model Element 
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Characterize SET 

Framework process with 
multiple subcontractors 



3.8 Modeling Methods Objectives 


Figure 3.5. Modeling Methods 
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Table 3.4. Modeling Methods Objectives 


Model Element 

Documentation 

Characterize MDAO 

Modeling Method 

The method for Multidisciplinary Design, Analysis and Optimization 
(MDAO) has been demonstrated in the SERC RT-168 research for CONOPS 
and system-level modeling. This objective seeks to characterize this through 
the use of examples and demonstrations for the surrogate pilot, and will carry 
this out using Phoenix Integration ModelCenter for the SERC researchers, 
but the Surrogate Contractor has their own MDAO tools. 

Characterize Method for 
Tracing between Mission 
and System models 

Need to characterize how the surrogate contractor can extend the system 
model and trace to discipline/domain-specific models, which also traces back 
up to the mission model. Should this also trace to CONOPS or CONEMPS? 
In the context of the model management method, where should the 


35 






















































































Model Element 

Documentation 


traceability linkages be created? 

Characterize Mission 

Modeling Method aligned 
with Integrated Capability 
Framework 

The objective is to defined a mission modeling method that aligns with the 
Integrated Capability Framework. This characterizes the processes for 
Mission Technical Baseline and the Integrated Capability Technical 
Baseline. 

Characterize Model 

Management Method 

The objective is to define one or more model management methods that can 
support a new operational paradigm specifically focused on new approaches 
to contracting that can operation more like software change control. 

Characterize 

Modularization Method 


Characterize System 

Modeling Method 

This characterizes the system modeling method. The initial recommendation 
was to use OOSEM, but can this apply to all type of programs? 

Demonstrate Use of 

Semantic Technologies and 
Ontologies 

There are many objectives for using Semantic Technologies and ontologies 
for formalizing methods and to support cross-domain model integration 
through interoperability of ontologies data. In addition, this particular model 
element is based on the NASA/JPL IMCE ontologies. 


3.9 Real-time Collaborate in AST Objectives 

Figure 3.6. Real-time Collaboration in AST 
bdd [Package] Real-time Collaboration in ASTI Real-time Collaboration in AST^J 



Figure 3.7. Use Cases for Collaboration Environment and Authoritative Source of Truth 
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pkg [Package] Real-time Collaboration in ASTJ Use Cases for Collaboration Environment and Authoritative Source of Truth ]J 
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Collaboration environment with role-based access for all stakeholders 
Was/ls analysis from View Editor to track all changes in models 
Determine roles for Surrogate Contractor (SC) to make entries through 
View Editor or possibly MMS in AST to link to SC-side of AST 
Provide roles so that External Monitoring Organizations can view what’s 
happening, and 

Provide means for External Monitoring Organizations to create “own” 
sub-repository so that different organizations can make/link comments 
that are either public or private to any stakeholders 
How do we link ontologies to MMS for semantic reasoning about 
methods and tool interoperability across domains (**SERC research) 



Authoritative Source of Truth 



Mechanical Design Models 


Electrical Design Models 


Software Design Models 


Testing Methods & Models 


Analysis Tools 


Table 3.5. Real-time Collaborate in AST Objectives 


Model Element 

Documentation 

Characterize Approach for 
Integrating Government 

and Industry Environments 
in AST 

This will include, but not be limited to: 

1. Collaboration environment with role-based access for all 
stakeholders 

2. Was/ls analysis from View Editor to track all changes in models 

3. Determine roles for Surrogate Contractor (SC) to make entries 
through View Editor or possibly MMS in AST to link to SC-side of 
AST 

4. Provide roles so that External Monitoring Organizations can view 
what’s happening, and 

5. Provide means for External Monitoring Organizations to create 
“own” sub-repository so that different organizations can make/link 
comments that are either public or private to any stakeholders 

6. How do we link ontologies to MMS for semantic reasoning about 
methods and tool interoperability across domains (**SERC 
research) 


Characterize Process for 
Insight and Oversight in 
AST 
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Model Element 

Documentation 

Package Modeling 

Environment as part of 
Collaboration 

The concept is to use Docker in order to allow the development environment 
used to produce the Initial System Model to be shared with the Surrogate 
Contractor. 

Set up Collaborative 

Environment for AST 


Use of OpenMBEE MMS 
to Demonstrate Model 
Management 

A key driver for NASA/JPL developing OpenMBEE was to provide model 
management at a fine level of granularity and be completely tool agnostic. 
This should provide a means for providing details tracking of changes that 
can support a new operational paradigm for managing contracts. 


3.10 Decision Framework Objectives 


Figure 3.8. Decision Framework 

bdd package] Decision Framework [ Decision Framework | l 



Table 3.6. Decision Framework Objectives 


Model Element 

Documentation 

Characterize assessing 
value of KPP 

A possible Decision Framework research tool and method for assigning value to 
KPPs is being applied to different case studies on the ARDEC research task RT- 
168. 

Characterize change to 
SETR process 

This objectives investigate how the SETR process/guidebook/checklist can be 
refined or modified by being able to make assessments more objectively within 
models. There is research that has started an ontology from the SETR guidebook. 
Can this be part of the objectives measures? 

Characterize objective 

This is a new process to allow for continuous asynchronous decision making 
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Model Element 

Documentation 

measures for 

evaluating design 

maturity 

about a maturing design as opposed to the traditional monolithic events (e.g., 
SRR, PDR, CDR). It is unclear if the objective measure apply to Element 1 or 2. 


3.11 Source Selection Objectives 


Figure 3.9. Source Selection 
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Table 3.7. Source Selection Objectives 

Model Element 

Documentation 

Characterize a Model-Based Acquisition Source Selection 


3.12 Operations and Policy of Contracting Objectives 

Figure 3.10. Operations and Policy of Contracting 
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bdd [Package] Operations and Policy of Contracting [ Operations and Policy of Contracting ]] 
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This section needs to be refined 
like the other sections. 
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Table 3.8. Operations and Policy of Contracting Objectives 


Model Element 

Documentation 

Characterize how models are 
used contracting 

Investigate the potential 

Characterize process for 
feedback to Element 1 or 
Element 2 after source 
selection due to unachievable 
KPP 

The objective is to determine processes enabled by modeling and all 
association enabling technologies for contracting related feedback due to 
issues in the contract after source selection. 

Characterize scenarios for 
investigating Data Rights and 
Intellectual Property 

Consider looking at the document from the Aerospace Industry 
Association CONOPs. Who owns the different models? Recall the 
approach used by NAVSEA and Huntington Ingalls called the Product 
Data Model (PDM) that was presented at the 2016 Model Centric 
Engineering Government and Industry Day. 
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3.13 Visualization Objectives 


Figure 3.11. Visualization 


bdd [Package] Visualization! Visualization ]J 


This is not complete. Other eamples include 
Tom Sawyer used by NASA/JPL. 
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JSmallwood 


Table 3.9. Visualization Objectives 


Model Element 

Documentation 

Characterize Representations 
for Model-derived 

Specification 


Characterize Representations 
support by ISEE 

These are existing mechanisms that will support visualizations. 

Demonstrate Model-driven 

Specification and Artifact 
Generation 

The objective is to demonstrate the uses of model-driven specifications to 
support contracting as well as other artifacts to provide the appropriate 
view and viewpoints relevant to different stakeholders. 
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3.14 Surrogate Evaluation Objectives 


Figure 3.12. Surrogate Evaluation 
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Table 3.10. Surrogate Evaluation Objectives 


Model Element 

Documentation 

Characterize approach for 
allowing external 

stakeholder to evaluate 
surrogate pilot 

This needs to refine the idea that we can selectively allow external 
stakeholders permission to log in to the Authoritative Source of Truth 
repository and provide ongoing feedback to all facets of the approach used 
on various phases of surrogate pilot. 

Characterize how to capture 
lessons learned related to 
Executing the Surrogate 
Pilot 

We need to characterize how we are going to capture lessons learned during 
the execution of numerous phases of this pilot, including capturing 
(potentially) anonymously information from external stakeholders such as 
Industry and Govenment organizations other than NAY AIR. 


4 UC00: Ontologies and semantic web technologies 


This use case investigates the development and use of ontologies and more generally semantic web 
technologies (SWT) for reasoning about completeness and consistency across cross-domain models. 
These capabilities support enforcement of modeling methods and support for model integration 
through interoperability. We summarize some research findings and our efforts related to SWT in this 
section. For example, we have developed the Integration and Interoperability Framework (lolF), which 
has been used for preliminary demonstration to support this concept. We plan to look into the use of 
our research on the surrogate pilot. 
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There is increased interest in the topic of ontologies and SWT as awareness has increased significantly 
in the past two years. We believe SWT may be enablers for an AST, tool-agnostic approaches to 
methodology enforcement, and conformance that also support model integrity as reflected in Figure 6. 
This section summarizes some of the relevant efforts over the past year on this topic in addition to the 
description and examples in Section 3 that explains how we are using the NASA/JPL IMCE ontologies 
[87] in the surrogate pilot. It is also important to say that SWT is an enabler for capabilities such as 
computational intelligence (e.g., Artificial Intelligence, etc.), because they provide a means for 
representing knowledge. We see these capabilities coming to use in both the systems we build and 
deploy, as well as in the systems engineering systems we use to analyze and development systems 
moving forward. 


4.1 Challenge of Cross-Domain Model Integration 

We believe that organizations should take advantage of tool-to-tool integration when possible, but in 
working with our sponsors and interacting with industry and government organizations, this is not 
always possible or it can be challenging. The challenge of cross-domain modeling integration can be 
illustrated using the following example. While an aircraft may have thousands of objects, consider the 
relationships for a refueling value of a UAV, as shown in Figure 10. There is one object discussed in this 
example (i.e., Valve), however, there are many domains that bring in cross-domain relationships to that 
Value, along with other objects, such as: 

■ Mechanical Domain 

o Valve connects to a Pipe 

■ Electrical Domain 

o Switch opens/closes Value 

o Maybe using a combinations of hardware and software 

■ Operator Domain 

o Pilot remotely sends message to control Value 

■ Communication Domain 

o Messages sent through networks: 1) within the aircraft system, and 2) from the remote 
operator 

■ Fire control Domain 

o Independent detection to shut off Valve 

■ Safety Domain 

o Looks top-down at potential hazards through Fault Tree Analysis (FTA) 
o Looks bottom-up using Failure Models and Effects Analysis (FMEA) to analyze failure impacts 
from specific designs of components 
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Figure 10. Example of Cross Domain Relationships Needed for System Trades, Analysis and Design 


A problem is understanding the cross-domain impacts of designs and analyses that might be needed if 
one object within these related domains change. In general, there are different tools used in different 
domains, and the tools are often not integrated, nor are they able to share semantically-relevant data. 
Tool integrations are often dynamic consequences of customer requirements to continue improving the 
tools, thus the tools are constantly being updated, which further adds to the challenge of tool-to-tool 
integration. Tool integrations are not simply statically putting a certain set of tools together. Depending 
on the varying needs of tasks from particular stakeholders, the types of tools needed, their execution 
sequences, the interdependencies of data flow among them vary from case to case. In addition, the 
problem often gets worse when attempting to maintain an integration for different versions of tools. 
Figure 11 illustrates the dynamic nature of tool integration [144]. 
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Figure 11. Coordination Across Tools Based on User Story 

As shown in Figure 12 [51], there can be a very large set of tools that can be used to support analysis 
and develop the needed data and information across all of the domains. Notionally the Reference 
Technology Platform (RTP) [6] is the collective set of tools that an organization has in their inventory. 
Any specific program creates a RTP instance. A key challenge is integrating the assembled tools, 
especially when they may not have been created to be integrated, and equally important is that the 
methods for assembling and using these analysis workflows is largely in the heads of a few subject 
matter experts, as explained by our sponsors. Therefore, it is important that appropriate methods are 
applied to the selected tools that are assembled for use on a project or program. As a secondary 
objective that is being demonstrated as a leading-edge approach by NASA/JPL is to ensure models are 
created that comply with established modeling patterns that have been formalized using ontologies. 
We provided information on the NASA/JPL approach, which transforms the model information into a 
tool-neutral AST based on ontologies, and then uses standard SWT to apply checks to ensure 
completeness and consistency [86]. 
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Figure 12. Appropriate Methods Needed Across Domains 


4.2 Semantic Web Technologies and Ontologies 

Briefly, the SWTs are based on a standard suite of languages, models, and tools that are suited to 
knowledge representation. Figure 13 provides a perspective on the SWT stack, which includes extended 
Markup Language (XML) [115], Resource Description Framework (RDF) [163], Web Ontology Language 
(OWL) [162] (i.e., OWL2), the SPARQL Protocol And RDF Query Language (SPARQL) [163], and others. 
RDF can describe instances of ontologies - that is, the data for particular model instances, where OWL 
relates more to metamodels describing the class of information and relationships that can be 
characterized as RDF instances. The SWT was created to extend the current Internet allowing 
combinations of metadata, structure, and various technologies enabling machines to derive meaning 
from information, both assisting and reducing human intervention. This technology is generally 
applicable to many different applications, and we discuss a few in the following section. 
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Figure 13. Semantic Web Technologies related to Layers of Abstraction 


4.3 NASA/JPL Integrated Model Centric Engineering (IMCE) Ontology 

Our research is beginning to reflect through demonstrations and presentations some of the different 
uses of SWT and ontologies. The following figures have been taking from Model-Centric Engineering, 
Port 3: Foundotionol Concepts for Building System Models [87]. Figure 15 shows the IMCE ontology 
concept that is being evolved by NASA/JPL. Their process involves: 

■ Creating the foundational IMCE systems engineering ontologies [90] derived from modeling 
patterns (reflected in Figure 14), including: 

o Mission ontologies 
o Project ontologies 
o Analysis ontologies 

o The rationale underlying these ontologies are currently being documented by NASA/JPL's 
Steve Jenkins are part of a new effort called the Semantic Technologies for Systems 
Engineering Foundation [151] 

■ The ontologies can be created with any OWL modeling tool such as the open source Protege 
[128] 

■ The ontologies are transformed into SysML profiles [89] 

■ The SysML profiles are loaded into a modeling tool (MagicDraw in this case) for creating models 

■ The profiled SysML models are exported back into OWL statements 

■ Checks for completeness, consistency and well-formedness can be performed 
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Partial Map of Foundation Ontology Concepts 


Model-Based Systems Engineering 


(animated) 



Figure 14. NASA/JPL Foundational Ontology for Systems Engineering 


Semantic Technology that is Modeling-tool independent Domain Specific Modeling Language (DSML) 
for Systems Engineering through Stereo Typed SysML 



Figure 15. From Ontologies to SysML Profiles and Back to Analyzable OWL / RDF 


Figure 16 shows the various representations associated with the concept described in Figure 15: 

1. The modeled statement in English is: "Component performs Function" 

2. The OWL/RDF representation of the statement in low-level XML for this same statement 
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3. The Profile and Stereotypes used in the model (loaded into a SysML model) 

4. The Stereotypes used in a SysML Block Definition Diagram (BDD) - while SysML is the graphical 
language that is used, the stereotypes derived from the ontologies effectively make the use in 
SysML into a Domain-specific Modeling Language 

(animated) 

English OWL * SysML Profile Usage 


Model-Based Systems Engineering 
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Figure 16. Multiple Representations in Process 


Figure 17 provides another perspective showing where these SWT would reside architecturally using an 
instantiation created by NASA/JPL of OpenMBEE. A number of the components we are interested in 
using, include: 

■ The NASA/JPL IMCE ontologies for systems engineering, highlighted in the center of the figure, 
are used to check constraints (e.g., consistency, completeness, well-formedness) [86] [87] 
related to the model as shown in Figure 14 

■ Three core elements of OpenMBEE are: View Editor, Model Development Kit (MDK)/DocGen 
and Model Management System (MMS); note OpenMBEE can be used without the JPL-IMCE 
ontologies [90] 

■ MagicDraw client (in which the MDK/DocGen) plugin works 

■ Teamwork Cloud server from NoMagic is used with MMS 
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Figure 17. NASA/JPL Instantiation of OpenMBEE (circa 2014) 


4.4 Digital Environnent at Airbus Space 

We have discussed the importance of an underlying information model (e.g., ontology) to enable the 
cross-domain integration of information in an AST [23]. The concept of semantic analysis that is 
integrated within the Integrated Modeling Environment (IME) is not limited to NASA/JPL. Ralf 
Hartmann, the Vice President of Enterprise Digitization for Airbus, gave a presentation at the NASA/JPL 
Symposium and Workshop in Jan 2017 [75]. While there were many points made in this presentation, of 
particular interest was a historical perspective on how they have been assembling a system design 
engineering environment to cover the entire lifecycle. The representation of the environment as shown 
in Figure 18 was particularly interesting as it relates to the concept of a semantically rich information; 
this pertains to the box in the middle call RangeDB Data Management. This is a relatively recent 
development where they replaced a commercial product with their own infrastructure functionality 
(i.e., "secret sauce") that provides a Semantic Data Model for multi-disciplinary Integration as shown in 
Figure 21. We did discuss this with a person from Airbus at the event, and asked about the strange 
name (i.e., RangeDB), and he said it was "historical." This effort confirms why we believe SWT will play a 
key role to characterize the underlying information model for both ARDEC and NAVAIR, and again 
reflects positively on the NASA/JPL use of SWT as discussed in this section. 
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Figure 18. Airbus Digital End-to-End (System & Product) Engineering 


Finally, the Hartmann briefing also included an associated roadmap as shown in Figure 19 that was 
structured in two dimensions: 

■ Technology clusters 

o Requirement engineering & V&V 
o MBSE and design 

o Engineering data lifecycle management 
o Collaborative engineering 

■ System engineering technology integration levels 
o Data integration (just connecting data) 

o Semantic integration (identifies rules how to connect and understand data) 
o End-to-end (knowledge management) 

The key reflection on this roadmap is acknowledging the increased need to formalize the underlying 
information model as we move to the right (i.e., future), which can exploit more computational 
automation such as computational intelligence (i.e., Al, machine learning, etc.), enabled by high 
performance computing. 
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Figure 19. Airbus Roadmap Shown Bands of Digital Engineering Integration 


4.5 Integration and Interoperability Framework (IoIF) 

Our instantiation of an Integrated Modeling Environment (IME) to support what has been discussed in 
the context of the use of SWT by JPL-IMCE within OpenMBEE and the Semantic Information Model of 
Airbus's IME is IoIF. This is an evolving part of our research, but we have executed demonstrations of 
IoIF as reflected in Figure 20. This example of the IoIF uses two active models and passes published data 
through the SWT layer before delivering the data to the subscribing model. The published data that is 
passed into the SWT is extracted in different units and by different name. The example demonstrates 
the ability of the IoIF to convert both units and name, through the following steps: 

■ SysML model includes model with a specific instance of linear speed 

■ DocGen transforms SysML model data to xml format 

■ Proxy A captures and transforms xml data to RDF 

■ Proxy A publishes linear speed (in m/s) to Data Acquisition and Aggregation (DAA) 

■ Linear speed variable name and units will not match what is needed for Proxy B 

■ Mission Model Proxy B subscribes to red team linear speed 

■ DAA handles publish and subscribe from proxies 

■ SWT resolves the differences in the variable naming of linear speed and also the units 

■ When Proxy A (DocGen) publishes a new linear speed then the DAA initiates a request to the 
SWT to get the needed information for the subscribers of that data (Mission Model) and sends 
the updated information to the subscriber (Mission Model) 

■ DAA stores RDF instance data 
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Figure 20. Integrating System Model Data through SWT to 2D Simulation 


Evolving updates on lolF, as reflected in Figure 21 continue to focus on SWT, including, relationships 
between OpenMBEE, Model Management System (MMS), View Editor, in the instantiation of Docker 
[62], which are now running an Amazon Web Service (AWS) as reflected in Figure 22. This capability was 
initially rolled-out in mid-February 2018, and more information and its use will be provided in future 
technical reports. 
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Figure 21. Integrating and Interoperability Framework (lolF) 
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Figure 22. Elements of Authoritative Source of Truth - Including lolF, MagicDraw, MMS, View Editor, Teamwork 

Cloud and Various Software Tools 


4.6 Cross-Domain Integration and Natural Language Process of Requirements 

Drs. Mark Austin and Leonard Petnga from University of Maryland (UMD) discussed their work at a 
NAVAIR working session on Systems Integration Capability using Semantic Modeling of Requirements, 
Components and Architectures. Mark discussed the concept and showed the results of a demonstration 
for some sample requirements his student provided from NASA Goddard. Leonard discussed the use of 
SWT for decoupling of architecture and component domains. Leonard also provided an example to 
automate the generation and checking of valid component relationships. The increased focus on SET 
has caused us to stop on this investigation. 


4.7 Ontology Derived from System Engineering Technical Review (SETR) Handbook 

Dr. Mary Bone started the development of an ontology for the classes of information described in the 
SETR Process Handbook as shown in Figure 24. We have not continued this work, because we are 
unsure how the SETR reviews will be used under the new concept being explored in the surrogate pilot. 
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Figure 23. Class Structure for Ontology Derived From SETR Process Handbook 


4.8 Decision Framework related to Cross-Domain Integration 

Working with our ARDEC sponsors and collaborators, we have advanced the concept of the Decision 
Framework and demonstrated the technical feasibility of capturing needed input information from 
models. Dr. Matt Cilli is in the process of producing a journal paper so that the computational 
algorithms underlying the Decision Framework will become public domain, which should allow us to use 
this concept with NAVAIR and other research sponsors. 

The unique research approach is to use SWT to demonstrate the concept to both characterize the 
underlying data and information, as well as rules, and query language for processing and data exchange. 
Several briefings on SWT concepts and example uses have been provided in both working session and 
webinar sessions. 

We will investigate this as a basis for an objective approach to assess design maturity based on an 
ontological representation of the system using standards-based SWT. This will provide the means for 
assessing completeness and consistency across different models, developed using different languages 
and methodologies. This will leverage SWT for creating an ontology to demonstrate concepts for 
reasoning about conceptual models and design model maturity, which is tool neutral. 

Figure 24 provides a perspective on a standard systems engineering flow to illustrate where the 
Decision Framework fits into the overarching analysis workflow: 

■ CONOPS derived from simulation and gaming technologies are used to look at scenarios for 
trade space analysis of mission alternative 

■ "What" we want - requirements and constraints for a system of System of Systems (SoS) 
mapping back to the mission requirements 

■ "How" (1 or more) - designs to achieve the "What" 
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■ "How well" (usually many) to assess the "How" using analysis, testing, reviews and assessing 
how the design satisfies the requirements, given the constraints to achieve the mission concept 

■ The underlying Information Model (ontology) links the data or metadata from many different 
domains 

■ The Decision Framework, we believe can demonstrate how data from the information model 
can be used to populate the Decision Framework 
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Figure 24. Context of System Engineering of Challenge Areas 

As discussed in the next use case, we have developed using Phoenix Integration's ModelCenter [125] 
and MBSEPak, with SysML a way to formalize some of the inputs needed for the Decision Framework. 

5 UC01: Multidisciplinary Design, Analysis and Optimization (MDAO) 


This use case investigates various uses of Multidisciplinary Design, Analysis and Optimization (MDAO) at 
the mission, system and subsystem levels, which provides a means for continual assessment of trades 
(i.e., analysis of alternatives) to support KPP assessment; this also relates to representations within 
system models. This use case also investigates the methods to trace capabilities to the relevant design 
disciplines and perform cross-domain analyses through MDAO for problem and design tradespace 
analyses. In addition, to characterizing elements of the framework, cross-domain relationships, but also 
characterize the methods used to support MDAO in a tool independent manner. 

MDAO is an approach for calculating optimal designs and understanding design trade-offs in an 
environment that simultaneously considers many types of simulations, evaluations, and objectives. For 
example, when designing an air vehicle, there is typically a trade-off between maximizing performance 
and maximizing efficiency, where calculating either of these objectives require multiple disciplinary 
models (geometry, weight, aerodynamics, propulsion). MDAO prescribes ways to integrate these 
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models and explore the necessary trade-offs among the objectives to make a design decision. While the 
theoretical foundations of MDAO are well-established by academics, a number of barriers to practical 
implementation exist. Chief among these is the lack of model integration, which prevents designers of 
one subsystem from easily assessing how changing a design variable affects the results of other 
subsystems' models or simulations. 

As illustrated by some of the examples shown below, we can extract the key parameters in these 
various mission and system simulations. These parameters are fundamental to the MDAO workflows. 
We need to combine those parameters for different elements of a workflow, but we must also 
characterize our KPPs; for example, a surveillance UAV range or endurance (e.g. number of hours of 
flight) would be examples of possible KPPs. These KPP are modeled as the outputs from running the 
MDAO through different optimizations. The other aspect of the method involves identifying the 
constraints that must be characterized with respect to KPPs (i.e. outputs) with respect to selected 
inputs. 


5.1 MDAO Objectives 

The following provides more specific objectives for MDAO use: 

■ Assessing the impacts of individual design changes on system capabilities 

■ Supporting early-phase (conceptual design), system-level trade-off analysis using previous 
evaluation results from existing models 

■ Develop strategies to transform the contracting process so that requests for proposals (RFPs) 
can be designed more flexibly toward value-based (rather than target-based) design 

In pursuit of these objectives, the research activities entail: 

■ Develop generic multidisciplinary models of NAVAIR-relevant system examples, including 
analyses of the geometry, structure, aerodynamics, propulsion, and performance capabilities, to 
be used as an example case 

■ Explore using systems representations (e.g. SysML, Domain Specific Models) to map all inputs 
(parameters and variables) and outputs (objectives, constraints, intermediate parameters) 
among the individual models 

■ Conduct trade studies on the UAS design using established approaches and tools for MDAO, 
exploring different approaches, tools, and visualization techniques to most effectively display 
information and uncertainty for decision-makers 

■ Explore ways that previous trade study results on detail-phase product design can be useful 
toward new conceptual design of products with varying mission capability requirements 

■ Use the surrogate pilot to understand the barriers to implementing this type of MDAO, 
culturally and practically/theoretically 

■ Explore more general ways to map and coordinate subject matter experts (SMEs) and data, 
models, and meta-models for improved requirements setting for RFP or CONOPS, and value- 
driven design 

■ Explore the ways that MDAO and MBSE tools can work together 

One of the objectives of this project is to leverage the most powerful tools that are often used by 
industry as well as government organization. We have secured academic licenses to Phoenix 
Integration's ModelCenter [125]. Further, while research to date examines the use of MDAO at the 
systems level. We have received additional academic licenses to ModelCenter to investigate the use of 
MDAO at the mission and subsystem levels. However, based on the concept of the SET Framework, 
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MDAO analysis at the subsystem level will probably be carried out by industry that is developing the 
designs. 


5.2 MDAO Methods 

Using tools like ModelCenter, we have investigated, demonstrated and described methods for applying 
such tools, and also identified relevant research questions in the context of those advanced tools. For 
example, the steps for an MDAO method may be characterized as: 

■ Describe a workflow (scenarios) for a KPP (e.g., range, notionally similar to surveillance time) 

■ Determine relevant set of inputs and outputs (parameters) 

■ Illustrate how to use a Design of Experiments (DoE) and use analyses such as sensitivity analysis 
and visualizations to understand the key parameter to use with optimizations 

■ Illustrate Optimization using solvers with key parameters and define different (key objective 
functions - on outputs) to determine set of solutions (results often provided as a table of 
possible solutions) 

■ Use visualizations to understand relationships of different solutions 

A number of methods can be applied to formulate multidisciplinary optimization problems, develop 
useful surrogate models, and calculate optimal and Pareto-optimal solutions. Optimization problems 
can be formulated with a number of different objectives by converting some objectives to targets or 
constraints, summing the objectives with value-based and unit-consistent weighting schemes, or 
multiplying and dividing objectives by one another. Surrogate models are often used to quickly simulate 
the behavior of a more computationally-intensive simulation model, and some common methods 
include interpolation, response surface using regression models, artificial neural networks, kriging, and 
support vector machines. Finally, numerical optimization can be performed using a number of different 
algorithms and techniques, including gradient-based methods, pattern search methods, and population- 
based methods. For each of these, different techniques have been found to be more suitable to 
different applications, and part of this research directive will be to identify and demonstrate the best 
tools for this MCE architecture. 


5.3 Integrations with Related Tasks 

Through this project, and the creation of an MCE architecture that follows an AST and a consistent 
ontology, we investigate how to leverage MDAO techniques in the design decision-making process. A 
solid framework for MDAO can enable multi-objective optimization, showing product developers how 
different design objectives compete with one another. For example, we know that improving an 
objective like "minimize weight" typically requires a sacrifice in the objective to "maximize power." The 
magnitude of that improvement-sacrifice relationship, which often involves different units and requires 
human judgement to make a mission-appropriate decision, can be revealed by combining different 
simulation models, surrogate models, and optimization routines. As this may involve balancing a large 
number of objectives, one of the key challenges is in visualization of the results to enable informed 
decision-making. This fits into all five tasks of the project, as the entire information architecture must be 
built to support cross-disciplinary analysis, and specific tools and techniques can be integrated and 
tested at different stages of the transformation. 
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5.4 MDAO UAV Examples and Use Cases 


Examples and demonstration covering several of the objectives have been presented in several working 
sessions as well as several bi-weekly status meetings. We have four use cases: 

1. Developing MDAO workflows for KPP examples at system level 

2. ModelCenter integrated with a Graphical Concept of Operation (CONOPS) example using Unity 
gaming engine at the mission level 

3. Integrating MagicDraw SysML models with ModelCenter and MBSEPak for an underwater super 
cavitating modeling system 

4. ModelCenter and MBSEPak, with MagicDraw SysML to formalize the concept of an Assessment 
Flow Diagram, which is part of the Decision Framework and process [44] 

This section provides a summary of some of the evolving use of MDAO in our research. 


5.4.1 MDAO Example for Fixed Wing UAV 

The first demonstrated workflow shown in Figure 25 was developed using ModelCenter. This 
demonstration covered several aspects of the modeling objectives discussed in this section, including: 

■ Describe and execute a workflow analysis of UAS capabilities (e.g., range, velocity, and fuel 
consumption) 

■ Map relationships among parameters (inputs/outputs) in disciplinary models 

■ Illustrate use of Design of Experiments (DoE), sensitivity analysis, and visualizations to 
understand capability relationships/trade-offs 

■ Optimize using different solvers to find sets of Pareto-optimal solutions 

■ Take advantage of previous model analyses for use in early-phase design with new mission 
capability requirements 
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Figure 25. MDAO Example Workflow 


As shown in Figure 26, the Pareto frontier (Pareto optimal set) shows the trade-off between range and 
propulsion. The blue points show the Pareto frontier/non-dominated solutions. The Pareto frontier was 
calculated using a bi-objective optimization using NSGA-II algorithm to: 

■ Maximize range 

■ Maximize propulsion 

■ Given 5 design variables 
o Wing area (ft2) 

o Wing span (ft) 
o Altitude (ft) 
o Speed (knots) 
o Efficiency factor 

These results reflect on how much range one would have to give up in order to increase the propulsion 
by some amount. Based on the current set of equations characterized in the workflow, the sensitivity 
analysis shown in Figure 27 indicates that the wing area is the variable that exhibits the clearest trade¬ 
off. The wing span has the largest effect on range, but does not present a trade-off between these 
objectives. 
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Figure 26. Pareto Frontier (Pareto Optimal Set) Shows Trade-off Between Range and Propulsion 
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Figure 27. Sensitivity of Objectives to Design Variables 
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5.4.2 Extending the MDAO UAV Example 1 

Brian Chell is a PhD student working with Dr. Steven Hoffenson. Brian has produced a number of 
updates to the initial model. The efforts produced alternative workflows that leverage other types of 
solvers for different aspects of the problem including multi-physics problems. For example, one of the 
first steps looked at bringing SolidWorks into ModelCenter as shown in Figure 28. This provides a way to 
bring in detailed geometries to the analysis. 
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Figure 28. MDAO Workflow with SolidWords Computer Aided Design Model 

There were a few challenges with the more complicated geometries, as well as: 

■ Open-source geometry validity is questionable 

■ Model variables 

o Most SolidWorks files found so far do not import variables into ModelCenter automatically 
o We assume that we can set the variables within SolidWorks, but this might be more difficult 
because manually setting values may not align structures (e.g., wing connect to fuselage to 
meeting correct) 

■ More complex 

o Computations solver (e.g., CFD) take longer to run on the laptops provided to students 
This has led to the following investigations: 

■ Equation-based models derived from the model shown in Section 5.4 

o Uses DLR Institute's Unmanned Combat Air Vehicles (UCAV) [97] parameters 
o Model is fully operational 

o Based on weight fractions that are more scalable, and easier to change than DLR UCAV 
model 

o Model starting with payload weight vs. range vs. endurance tradeoffs 
o Looking at the potential to merge with future Computational Fluid Dynamic (CFD) results 
with Finite Element Analysis (FEA) 

■ Simulation-based models 
o Difficulties 
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• Still problems with importing variables into ModelCenter 

• Very large number of variables automatically imported (12,000+) 

• Under construction 

o Consider open source simulation OpenVSP [119] vs. Solidworks (CFD) 

• OpenVSP is a parametric aircraft geometry tool 

• OpenVSP allows the user to create a 3D model of an aircraft defined by common 
engineering parameters. This model can be processed into formats suitable for 
engineering analysis. 

• OpenVSP commonly used with ModelCenter 

• SolidWorks has stronger analysis capabilities 

• OpenVSP is limited to a standardized shape library 

• SolidWorks Flow Simulation can handle turbulence 

• OpenVSP CFD is most valid at nominal flight conditions (e.g. low angle of attack) 

• OpenVSP should be sufficient for conceptual design phase 

OpenVSP is being used for CFD. It is easier to use with limited library of shapes of quadcopters and fixed 
wing, and can run 'headless' (i.e., without GUI) to make computations less expensive. NASA has been 
using this with ModelCenter. The current status is: 

■ Integrated parametric geometry and CFD into ModelCenter 

■ Performing optimization and DOE to characterize model 

■ Trying to find lowest-fidelity mesh that produces accurate results 

■ Challenges: 

o Takes some time to change between different aircraft 
o Future NASA wrapper will make this much easier 

o High-fidelity CFD simulations are very slow on low-end laptops like those provided to 
students; need to determine if Stevens and provide higher performance computing 
resources 

Figure 29 show the CFD results from the same geometry under the same flight conditions with different 
fidelity meshes. The simulation on the left has a coefficient of lift many magnitudes higher than the one 
on the right. Investigate mesh balancing accurate results and low computing cost. 



Figure 29. CFD Mesh Fidelity Importance 

More recent updates include analysis for both CFD and FEA with the objective to maximize endurance 
and range, and minimize stress at every span-wise node. This is done with a new workflow as shown in 
Figure 30, with the resulting aircraft shown in Figure 31. 
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Figure 30. Update MDAO Workflow including CFD and FEA 
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without FEA 

Figure 31. Resulting Aircraft Designs with and without FEA 


5.5 MDAO at the Mission Level using Graphical CONOPS 

This use case investigates an extension of the prior work to using the Graphical CONOPS technologies 
Unity gaming engine with MDAO using ModelCenter. The MDAO methods used: 

■ Design of Experiments (DoE) to run the simulation over the entire range of every input variable 
o Choose an appropriate DoE sampling method to shorten run time 

• Full Factorial 

• Latin Hypercube 

■ Sensitivity Analysis 

o Find which outputs are most sensitive to which input variables 

o Can remove (or fix the value) of non-sensitive variables to save time during optimizations 

■ Optimization 

o Use algorithm to optimize desired objective(s) 

While there were challenges that were overcome, the experiment demonstrated that it is possible to 
use MDAO to optimize for mission success, and the number of experiments (runs) to cover the DoE 
space of 1000s cases versus 10s of cases that would be covered by running the scenarios manually. 
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Figure 32. Explore the Integration of Graphical CONOPS Simulation with MDAO Tools 

The capabilities covered focused on objective to understand and overcome the challenges for a fully 
automated MDAO at the Graphical CONOPS level, including: 

■ Performance is measured by degree of success of a mission 

■ Artificial Intelligence (Al) is applied to counterparties so that they can adapt to and learn 
behavior of system 

■ Full automation - there was no humans in the loop, except for validation of behavior 

■ Simulated environment that includes counterparties was observed to behave in a surprising 
manner (e.g., there was emergent behavior) 

■ Software communicates programmatically through file transfer - as opposed to being directed 
manually 

■ Monte Carlo results in thousands of runs (vs. 10s when run manually) are made for each initial 
state to provide statistics 

■ Simulation can run at high speed to maximize statistics and in real time to allow for human 
validation of simulation behavior 

The finding suggests that MDAO can be used to optimize for system-level mission success to study far 
more trades than can be performed manually. The initial attempt created the simulation and removed 
the CONOPs visualization using a "headless" simulation that is wrapped by ModelCenter. Initially the 
architecture of the simulation was not enabled to operate in batch modes, and therefore the software 
had to be re-written to work with ModelCenter. When the simulation is running, the human cannot 
make edits, but the re-written and wrapped simulation can run thousands of design of experiments 
(DoE). The initial simulation ran in real-time, but a recent update now can run faster than real-time. 
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A time-step analysis for the new design that can run faster than real-time for the current three missions 
suggests: 

1. The time-step is very dependent on the timescales and complexity of the mission. 

2. We have initial measure of the quality of a run through the measurement of the average kinetic 
energy of the blue drone. The simulation fails when this number suddenly drops. 

3. For our current missions, we can increase the physics time step by about a factor of two (2). This 
means a speedup of about a factor of two (2). This is a smaller number than was expected going 
into the study. 

4. In physics simulations, one is often not interested in high-frequency behavior because we are 
interested in long-term bulk behavior of matter. In our case, we are very interested in high- 
frequency behavior because that behavior is used to determine the response of the agents to 
each other. This is like a basketball game in which players have head fakes and tells on short 
timescales. The defense must pick up on the short timescale events in order to respond on 
longer timescales. 

The proposed next steps include: 

1. Check the sensitivity of the speed-up with other missions. The current red drone strategy is to 
pursue the blue drone and interfere with it. 

o What if the red strategy was more of a zone defense? 
o What emergent surprises will we see? 

o We saw unexpected flocking behavior in the current red strategy? 

2. This red strategy is very different than the one we currently have. How does this affect the 
speed-up? 

3. Find a better measure of performance than the average kinetic energy. 

4. Incorporate this into the output file so that it is convenient analysis and programmatic 
processing. 


5.6 SysML Integration to MDAO through MBSEPak 

This research investigated the use of the Phoenix Integration MBSEPak (formerly MBSE Analyzer) that 
provides a way to integrate MagicDraw SysML models with ModelCenter for performing MDAO 
analysis. Dr. John Dzielski who performed this research primarily works in Matlab, and he used an 
example that was familiar to him related to underwater super cavitating modeling. The process covered 
the following steps: 

■ Defining requirements models in SysML 

o MBSEPak works by adding a profile that includes a number of stereotypes to MagicDraw 
o Specify a constraint (='s), upper and/or lower bounds, and units 

■ Connect properties to requirements via the satisfy relationship 

■ Information is transferred to ModelCenter through MBSEPak plugin as shown in Figure 33 
o Requirements are shown in the Margin column of the plug-in. 

o The plug-in indicates whether the requirements are satisfied or not by a design 

■ MagicDraw Plug-In populates an analysis to create a workflow 
o Components correspond to constraint blocks 

o Constraints blocks are models or equations used in par diagrams 
o Constraint parameters correspond to component variables in ModelCenter 

■ Parametric (PAR) blocks are used to indicate to ModelCenter how to connect component I/O 
(values) to model values 
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All of the other types of analyses discussed previous can then be applied in ModelCenter 



Figure 33. Example of MBSE Analyzer MagicDraw Plugin to Integrate with ModelCenter 

The following reflects on some of the initial findings in his first exposure to MagicDraw, SysML, and 
ModelCenter: 

■ SysML is best used after some training 

o Even for someone skilled at modeling in Simulink, SysML has a lot of documentation, but 
MagicDraw can be hard to learn (but he was able to do it, on his own) 

■ ModelCenter is a little bit easier to understand without any training 

o Extremely flexible, anything that can be modeled in ModelCenter can be used as a 
constraint 

o Similar constraints will be found in other types of weapon systems 
o MBSEPak works by adding a profile that includes a number of stereotypes to MagicDraw 

■ It is easier to model in SysML and use the MBSEPak than to create the ModelCenter workflows 
manually 

■ ModelCenter does not understand generalization relationships as represented in SysML 


5.7 Formalizing Assessment Flow Diagrams as MDAO Workflow 

For populating the Decision Framework [44] as discussed in Section 4.8, we need to collect all of the 
elements of information. The research objective is to determine how/where to collect all of the 
information reflected Figure 35 from rigorously specified models. Based on inputs from Dr. Matt Cilli, 
some of the underlying computations are going to be published in a journal paper. This would allow us 
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to perform most of the computation directly on the data stored in a triple store, and then extract 
information directly for the visualization. Matt is using this approach with the research affiliated with 
the Engineered Resilient Systems effort and created the visualization using Tableau software. This 
would provide senior leaders and program managers the type of information they need to consider 
technology capability tradeoff using Performance, Cost (Affordability), Time (delivery schedule) and 
Risk, as shown in Figure 34. 
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Figure 34. Visualizing Alternatives - Value Scatterplot with Assessing Impact of Uncertainty 


Fundamentally, if a particular answer was unacceptable, using the concept discussed herein, we could 
trace linkages through the Information model back to all other related perspectives on the system in 
terms of operational, mission, system, and subsystem design alternatives and trades. These elements 
would include: 

■ Objective hierarchies 

■ Value functions 

■ Assessment Flow Diagrams (AFDs) trace the relationships between physical means, intermediate 
measures, and fundamental objectives 

■ Uncertainties 
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This research used a case study documented in Matt Cilli's book chapter. We focused on formalizing the 
AFD using SysML, which is usually done in PowerPoint, as shown in Figure 36. This research investigates 
if we can formalize the AFD in SysML and be transformed into an MDAO workflow. Similar to the 
approach discussed in Section 5.6, we started with SysML and used the MBSEPak to produce the MDAO 
workflow, as reflected in Figure 36, which provides a basic conceptualization for researching this 
concept and to address the questions: 

■ Can MDAO represent Assessment Flow Diagram? 

■ Does AFD characterize needed MDAO workflows? 
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Figure 36. Formalizing the Assessment Flow Diagram 


The current results have formalized the representations of AFD using SysML, MBSEPak and 
ModelCenter, because the Key Performance Parameters can be mapped to one or more MDAO 
workflows as reflected in Figure 36. With some recommendation on modeling best practices for using 
MBSEPak with SysML from Phoenix Integration. A Webinar explaining this approach is provided at the 
Phoenix Integration website ( https://www.phoenix-int.com/learn-more/webinars/) called "Applications 
for Three Research Use Cases in Model Centric Engineering using ModelCenter and MBSEPak." [21] 

The modeling steps follow from the Decision Support Construct: 

1. Model system structure in SysML 

2. Model as derived value types in SysML decomposition 

3. Add the needed Measure scorecard that contains the Metrics of interest in the analysis 

4. Value scorecard provides basis to compare metrics as perceived by user 
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Figure 38. MBSEPak Creates Analysis Workflow and Checks Data Type Consistency 
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6 UC02: Integrated Modeling Environment (IME) 


This use case investigates topics for Integrated Modeling Environments (IMEs) with specific focus on 
creating and collaborating in an Authoritative Source of Truth (AST) for the surrogate pilot in the 
context of the research thrusts. This includes, but is not limited to: 

■ Formalization of the collaboration using an AST, in the context of NAVAIR, but also in the 
context of one or more industry contractors 

■ Model visualization from multiple perspectives including, but not limited to enabling different 
views relevant to different stakeholder (or due to particular access), reducing complexity, and 
analytical analysis 

■ Methods for model modularization to ensure separation of concerns, classification, and 
acquisition 

■ Methods for creating and organizing Enterprise, Process, and Reference models 

■ Understanding the operational paradigm between industry and government in the context of 
the SET Framework through MCE 

■ Workflow analysis and representation relative to a program instantiation of tool suites from the 
IME 

■ NOTE: cyber security and classification is not currently in the scope of this work 


6.1 Canonical Reference Architecture for Integrated Modeling Environment 

Our prior research suggested that the following types of capabilities characterized as a canonical 
reference architecture of an Integrated MCE Environment represent the aggregate of what is in use by 
different organizations across industry and government [9] [10] [11] [43] [57] [75] [80] [118] [86] [127], 
as shown in Figure 39. The following provides some perspectives on these general capabilities: 

■ Provides appropriate views for the various stakeholders 

■ Stakeholders have views into the Single Source of Truth (SST) 

o We think that there can be a SST, but in the context of a broader ecosystem required for 
model-based acquisition, where the government and generally multiple contractors have 
their own environment that the notion of an Authoritative Source of Truth (AST) may be 
more appropriate 

o Using rich modeling interfaces for those with expertise in modeling 

o Using rich "web" interfaces, which today provides support for graphics, integrated with 
structure inputs, generated textual views and 3D model viewing [132] 

■ MDAO layer provides for problem and design space exploration of 
o Physics-based models 

o Integrity-based models 
o Cost and scheduling models 
o Risk models 
o Various "illities" models 

■ Includes surrogates and components 

■ Enabled by High Performance Computing (HPC) 

■ Semantically rich linkages between data and information in the AST provides for continuous 
workflow orchestration between government and contractor(s) - enabled by more HPC 

■ Document generation is enabled by 

o Semantically rich links to information in the AST 
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o Templates that formalize patterns for requirements, contracts, etc. 

■ Enabling technologies such as machine learning provides a virtual knowledge librarian that assist 
users guided by embedding knowledge and training 

■ Contractor and collaborators have a secure means to plugin to view or share digital information, 
as a new paradigm for interactions 

■ This view of the Designing System provides links downstream to fully link Product Lifecycle 
Management (PLM) 
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Figure 39. Integrated Environment for Iterative Tradespace Analysis of Problem and Design Space 


We also believe that in the context of the SET Framework, these capabilities may be distributed 
between government and industry coupled by the AST. Our surrogate pilot will hopefully provide 
additional insight into those beliefs. 


6.2 Integrated Modeling Environment for Surrogate Pilot 

Given the context about IMEs, we have been able to assemble some relevant tools in the Stevens 
laboratory, but also deployed an instantiation of OpenMBEE to an Amazon Web Service server to 
support the surrogate pilot, which notionally will contain elements shown in Figure 22. We anticipate 
the surrogate contractor to have some of the same tools (e.g., MagicDraw), but additional tools that 
they have developed for MDAO, model-based design and collaboration. The objective will be to define 
the operational paradigm by which we look at these linked pieces of model elements (from the SET 
Framework) into an AST. The OpenMBEE is now operational can be quickly installed using Docker [62]. 
The current research thrusts are to integrate one or more environments to establish a distributed AST. 
We also need to define and characterize the needed capabilities of lolF, given that we will leverage tool- 
to-tool integration (e.g., Magic Draw and ModelCenter). We believe that these capabilities may be 
similar to those discussed as RangeDB in the context of the Airbus environment shown in Figure 18. 
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The lolF provides a framework to effectively interchange information using SWT and data acquisition 
functionality as shown in an example in Figure 40. This framework is being designed to be used with 
various software tools and various simulation environments. Each tool or simulation would have a Proxy 
to perform Publish (send information) and Subscribe (receive information) functions. The DAA layer 
would also interact with SWT of integrated ontologies to computationally interrelate and transform 
information such as the example discussed in Section 4.5. The immediate goals are: 

■ Abstracts away from the software client tools as much knowledge and dependencies of the tool- 
to-tool data integration architecture as possible 

■ Allows for tool-to-tool data integration on computer systems that are physically remote from 
each other 

■ Uses an ontology framework (i.e., SWT) that implements an automated decision process 
regarding tool/data relationships 

■ Uses a Publish / Subscribe framework that implements an automated data transport layer 
between various software client tools 

■ Investigate how to integrate with OpenMBEE 



Figure 40. Tool-to-Tool Integration and Interoperability Framework 

7 UC03: Modeling Methods 


This use case investigates the development and demonstrations of methods for technologies in the 
context of the IME workflows, such as: 

■ Methods for mission model 

■ Methods for system model 

■ Methods for modularizing models to support constraints needed for developing an authoritative 
source of truth, which relates to many other use cases 

■ Methods for model management 
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■ Methods for representing and organizing reference models, process models, discipline-specific 
models 

■ Methods for MDAO modeling are discussed in Section 5 

■ Methods for traceability 

■ Alternative approaches to improve modeling methods, which is fundamental to ensuring model 
integrity 


7.1 Mission Model 

The approach for developing the mission model is based on an Integrated Capability Framework (ICF) 
Operational Concept Document (Version 3.2) 22 February 2016. This document is considered 
"Distribution D," which means it may only be available to companies that are doing business with the 
government. However, the Skyzer Mission model will be made available publically on the Amazon Web 
Services server. This approach demonstrates that modeling can be used and comply with existing 
standards that traditionally have been document-based. 

The guidelines include: 

■ Thoroughly define required mission capabilities, measures of effectiveness, and associated 
operational conditions and constraints. 

■ Identify System of Systems (SoS) interfaces and measures of performance through structured 
decomposition of required mission capabilities. 

■ Provide a common, cross-Systems Command (SYSCOM)/Program Executive Office (PEO) 
framework to facilitate enterprise level engineering across the SYSCOMs and enable efficient 
system integration and effective force interoperability. 

■ Establish enterprise data structures and implementation guidance to enable iterative 
development of enterprise architectures 

■ The consistent implementation of ICF practices and guidance across assessments and 
stakeholders supports: 

o A common understanding of mission requirements and a structured process to identify and 
align systems and platforms capabilities to support missions, 
o System and platform owners with a thorough set of interoperability requirements and 
knowledge of what platforms, interfaces and behavior to which they need to design, along 
with associated standards. 

We have a View and Viewpoint hierarchy that extracts information from the Skyzer Mission model to 
"generate a specification," which aligns with the guidelines of the ICF. A portion of the View and 
Viewpoint hierarchy is shown in Figure 41. 
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Figure 41. View and Viewpoint Hierarchy for Surrogate Pilot Mission Model 


7.2 System Model 

NAVAIR decided to adopt Object-Oriented Systems Engineering Method (OOSEM) [98] as the default for 
System Modeling using SysML. There are many resources available that describe OOSEM. The main 
activities have been captured as a reference model. An example of is shown in Figure 42. 
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Figure 42. OOSEM Top Level Activities 


7.3 Modularizing the SysML Model 

The method for modularization models is also an important part of our surrogate pilot effort. As shown 
in Figure 43, we are using an approach for modularizing the surrogate pilot model that uses a "model 
reference" (Project uses) concept so that the mission, system and other models can be created 
independently, but could be referenced in an overarching project/program model. For example, in the 
containment tree on the left side of Figure 43, there are some packages, (e.g., Enterprise, Reference 
Models) that are in normal black font, but two models the Mission Level and System Level are slightly 
"grayed out," because these projects are references to separate models. In doing this, we can allow the 
Mission model and System model to be worked on separately, but when brought into the higher-level 
project model, we could view the entire model. In addition, as shown in the View and Viewpoint 
hierarchy, we can include these referenced models in one or more Views with Viewpoints, where 
DocGen can then generate a document or specification for the entire project or a subset of elements 
from various models. This concept of modularization would apply to other process models, such as 
those developed by competencies and reference models. We are investigating this evolving method, 
because it plays heavily with model management including tradeoff for both the Teamwork Cloud and 
OpenMBEE MMS. 
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Figure 43. Modularizing Surrogate Pilot Model 


7.4 Views and Viewpoints 

The basic elements, as shown in Figure 44 can be included within an overarching document, which 
includes: 

■ Document - the overarching model element 

o Document can include other documents, which also provides another level of 
modularization and support for reuse 

■ View (there can be one or more views in a document) 

■ A View uses the Exposes relationship to associate the View with some element in the model 
(e.g., Package, Diagram, etc.) 

■ View conforms to a Viewpoint 

■ Viewpoint in Model Development Kit (MDK) is a special language created out of a profiled 
activity diagram that can collect, filter, and then produce a document through a DocBook 
standard 
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Figure 44. Element of View and Viewpoints 


A document assembled from a number of Documents or Views can be generated into DocBook, which 
can then be generated into PDF, Word, HTML, and other formats. However, these Views can also be 
"pushed" into the OpenMBEE Model Management System (MMS) as shown in Figure 45. The View 
Editor can then be used to view the generated specification; in addition, it can export (generate) into 
Word, PDF, and HTML. The View Editor also allows for editing and updating a generated view that can 
also be pushed back into the MMS, as well as back into the model (for certain types of model elements). 



View Editor: Provides Rich Web Interface 


Figure 45. Views are Pushed into Model Management System and Viewable through View Editor 

As shown in Figure 46, the View Editor runs in a standard browser and lets users navigate the View 
hierarchy, and visualize specific Views within the hierarchy, edit the views and examine history 
associated with changes of the View. There are capabilities for branching those changes. This is part of 
the future research to investigate the combination of facets related to View and Viewpoint hierarchies, 
model management in MMS as well as in Teamwork cloud. We are expecting some support from 
NASA/JPL who is developing some type of guideline, and working in conjunction with our NAVAIR 
sponsors on the best methods for model management. 
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7.5 Methods for Traceability 

SysML and most of the modeling tools provide means for capturing traceability information, including 
various standard stereotypes. Our concern is tracing between the model levels, from mission to system 
to design models, and linking back to the KPPs. This is yet another objective that is defined in the 
surrogate project plan, but we have not gotten deeply enough into the modeling to characterize how 
best to support linking models at different levels. This is part of the research to be continued under the 
new RT-195 research task. 

8 UC04: Model-physics modeling and Model Integrity 


This use case investigates model-physics modeling, MDAO and model integrity which is also supported 
by MDAO and approaches for assessing model integrity risks and uncertainty. Model integrity, from our 
sponsor's perspective, is a means to understand margins and uncertainty in what models and 
associated simulations "predict" or in other words when/how do we trust the models. The objectives 
characterized by the sponsor are to ensure that the research covers the key objectives, which included: 

■ Include both models to assess "performance" and models for assessing "integrity" such as: 

o Performance: aero, propulsion, sensors, etc. 

o Integrity: Finite Element Analysis (FEA), Computational Fluid Dynamics (CFD), reliability, etc. 
- can we build it, can we trust it 

o A stated challenge was: how can "integrity" be accomplished when the current situation 
involves federations of models that are not integrated? 
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■ Continuous hierarchical and vertical flow enabled by models and iterative refinement through 
tradespace analysis, concept engineering, and architecture and design analysis 


8.1 Surrogate Pilot Design Model Constraint 

We have imposed constraints on the mission scenarios for the surrogate pilot to ensure that we have 
the opportunity to evaluate multi-physic designs and measures for understanding model integrity to 
support a production readiness decision. During Elements 1 and 2, we would use MDAO type analysis 
such as described in Section 5.4. The more critical aspects that concern our sponsor are the ability to 
deal with designs in Element 3, that can support a producability decision associated with Element 4 
when multi-physics design elements are involved in the decision process; that is, can we make a 
production decision from various type of modeling and simulation analyses of a design. An example is 
shown in Figure 31, which shows that there can be significant differences in the system design 
tradespace when both CFD and FEA are used in the same MDAO workflow. Therefore, this is another 
key objective of the surrogate pilot. The objective is to define mission use cases that can be used to 
force analysis to better understand the feasible multi-physics design options. 


8.2 Advanced Approaches to Model Integrity 

It is currently unclear if NAVAIR, in the context of the SET Framework, will ever deal with multi-physics 
consideration during Element 1 and 2 of the framework. Most of the analysis will likely be parametric in 
nature during Element 1 and 2. However, we do know that Sandia National Laboratory has discussed 
some of the most advanced approaches for supporting uncertainty quantification (UQ) to enable risk- 
informed decision-making [111]. Their methods and tooling address the subjects of margins, 
sensitivities, and uncertainties. The information they provided reflects on the advanced nature of their 
efforts and continuous evolution through modeling and simulations capabilities that operate on some 
of the most powerful high-performance computing (HPC) resources in the world. We heard about their 
HPC capabilities, methodologies on Quantification of Margins and Uncertainty (QMU), an enabling 
framework called Design Analysis Kit for Optimization and Terascale Applications (DAKOTA) Toolkit 
[141], and the need and challenge of Model Validation and Simulation Qualification [138]. They also 
discussed the movement towards Common Engineering Environment that makes these capabilities 
pervasively available to their entire engineering team (i.e., the designing system in our terminology). 
We think their capabilities provide substantial evidence for the types of capabilities that should be part 
of the risk framework. This section provides additional details. 

Traditional approaches referred to as Verification, Validation and Accreditation (VV&A) of modeling and 
simulation capabilities are still relevant and used by organizations. VV&A, in principle, is a process for 
reducing risk; in that sense VV&A provides a way for establishing whether a particular modeling and 
simulation and its input data are suitable and credible for a particular use [61]. The words "tool 
qualification" [63] and "simulation qualification" [138] have also been used by organizations regarding 
the trust in models and simulations capabilities. A more extension discussion of this subject is provided 
in RT-141 [28] and RT-157 [29]. 

9 UC05: Representation to formalize Monterey Phoenix for requirement 

VERIFICATION AND VALIDATION 


This use case investigates the development of SysML representations to formalize the Monterey 
Phoenix (MP) research under RT-176 to support requirement verification and validation [70]. MCE does 
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provide some unique opportunity to be more effective at contributing V&V evidence in early design. 
Rigorously defined models can directly support V&V, and this could both subsume cost and risks. 


The results accomplished against this effort to use SERC RT-176 effort of Monterey Phoenix for V&V of 
requirements is showing progress. The basic concept is to formalize using SysML graphics, and in this 
case activity diagrams and then transform into the MP language as shown in Figure 47. MP then uses 
the formal language to generate graphical representations of the behaviors, as shown in Figure 48 [133] 
that can be derived from the language of the formalized behavior to a given scope level (e.g., Scope 2 in 
Figure 47). The verification step does require a person to check the different behavioral representations 
for correctness. This concept is similar to model checking. 


—Including: scope, 
multiple roots, 
loop, alternative 


Start 



2. Model 
transformation 


12 A, B SHARE ALL Multi; 


3. Event trace 
generation 


Figure 47. Representation and Transformation from SysML Activity Diagrams to MP 
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Valid Scenario: Object detected, tracked, and Invalid Scenario: Target tracked 

determined by Swarm Operator to be a valid target after bingo fuel condition 



Example Found Requirement: A UAV shall only track 
targets found before reaching bingo fuel conditions. 


Figure 48. Generated Visualization of Scenarios by Monterey Phoenix 

More information on Monterey Phoenix can be found: 

■ MP Public Website: wiki.nps.edu/display/MP/ 

■ MP Analyzer on Firebird: http://firebird.nps.edu 

■ Giammarco, K., Practical modeling concepts for engineering emergence in systems of systems, in 
12th System of Systems Engineering Conference (SoSE). 2017, IEEE. p. 1-6. 

■ Revill, M.B., UAV sworm behavior modeling for early exposure of failure modes. 2016, Naval 
Postgraduate School: Monterey, CA. 

■ (pending) Quartuccio, J. and K. Giammarco, A Model-Based Approach to Investigate Emergent 
Behaviors in Systems of Systems, in Engineering Emergence: A Modeling and Simulation 
Approach, L. Rainey and M. Jamshidi, Editors. 2017. 

10 UC06: Experimentation and learning for research topics in the execution of SET 


This use case defines the objectives for experimentation and learning in the execution of the SET. This 
section is already characterized in an evolving state in the automatically generated report from the 
Surrogate Pilot Project Model, which effectively address the uses cases shown in Figure 49, and are 
being characterized in Section 3.4. 
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11 Enterprise Transformation to support governance and workforce development 


This use case is for research into Enterprise Transformation to support governance and workforce 
development. This has not yet been funded, but will be if RT-195 is funded. This research will be 
supported by Dr. Donna Rhodes from Massachusetts Institute of Technology. 


12 SERC Research Synergies 


This section discusses some synergies to the ongoing NAVAIR research tasks that are briefly mentioned 
in this report to inform readers of the relationships to these other activities. 


12.1 RT-168 ARDEC RESEARCH 

The most significant research synergies are coming from the ARDEC research under RT-168. We do 
many types of event, meetings, demonstrations and discussions with ARDEC that include NAVAIR. We 
are using a Model Based System Engineering (MBSE) approach to model aspects of our project. We 
elaborate the research tasks using high-level use cases, relating those use cases, and associating the use 
cases with stakeholders involved in the research as shown in Figure 50. It should be clear that the use 
cases are related, and stakeholders (including some from ARDEC) are involved in multiple use cases. For 
example, both ARDEC and NAVAIR are interested in OpenMBEE. This bi-monthly status will be given 
primarily in the context of each use case, and not necessarily in terms of stakeholder contributions, 
because some efforts have several contributors. 
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Figure 50. High-level Research Use Cases 


12.2 RT-176 Verification and Validation (V&V) of System Behavior Specifications 


Our NAVAIR sponsor had requested that the SERC RT-176 research task being led by Dr. Kristin 
Giammarco be aligned with the ongoing research from RT-157 and RT-170. This research synergy is 
discussed in Section 9. 


12.3 Aerospace Industry Association CONOPS for MBSE Collaboration 

This is a follow-up to the effort completed last year which developed a white paper on the Life Cycle 
Benefits of Collaborative MBSE Use for Early Requirements Development [3]. This white paper discusses 
the current state and benefits of MBSE across the entire life cycle and provides proposals for addressing 
such issues as MBSE Collaborative Framework, Government Data Rights, Intellectual Property, and Life 
Cycle Effectiveness with MBSE. 

The effort for this year involves many of the industry contractors to NAVAIR and DoD. The results 
should produce a white paper describing a CONOPS for how industry and government can collaborate 
through MCE/MBSE. 


12.4 OpenMBEE and Open Collaboration Group for MBSE 

We recently joined the Open Collaboration Group for MBSE that is providing support for adopting and 
contributing to OpenMBEE [118]. We are planning to use OpenMBEE in our lab, and contribute to the 
community effort in order to advance it with capabilities developed under RT-168, RT-170 and RT-176. 
We have been invited to present our work on OpenMBEE and its relationship to the Surrogate Pilot to 
this group in April 2018. 
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12.5 Semantic Technologies Foundation Initiative for Systems Engineering 


The NASA/JPL Symposium and Workshop on MBSE had a keynote talk given by Steve Jenkins that was 
fundamentally based on SWT and a foundational ontology for Systems Engineering developed my 
NASA/JPL. There were also two breakout sessions on the subject SWT. There was significant attendance 
at the break out session titled: "Ontologies, Formalisms, & Reasoning" possibly due to the motivation 
given by Steve Jenkins. In general, there is progress being made in this area and there is significant 
interest. Dinesh Verma has initiated an effort with the support of Chi Lin, Steve Jenkins and Mark 
Blackburn to bring a community of people together in an attempt to create and ecosystem on Semantic 
Technologies for Systems Engineering. 

The working group has created a charter and mission: 

■ Charter 

o The Semantic Technologies Foundation Initiative for Systems Engineering is to promote and 
champion the development and utilization of ontologies and semantic technologies to 
support system engineering practice, education, and research. 

■ Mission 

o The mission of the initiative is to collect a suite of interoperable ontologies that are logically 
well-formed and accurate from both scientific and engineering points of view. The initiative 
will charter a collective of stakeholders that are committed to collaboration and adherence 
to shared semantic principles for the advancement of systems engineering. To achieve this, 
initiative working group participants will voluntarily adhere to and contribute to the 
development of an evolving set of principles including open use, collaborative development, 
and non-overlapping and appropriately-scoped content. They will capture and maintain 
metadata for each ontology to encourage implementation and reuse. 


12.6 National Defense Industry Association Modeling and Simulation 

National Defense Industry Association (NDIA) Modeling and Simulation group is looking at approaches 
for using digital engineering for competitive down select. We are involved in all of these efforts to 
further the objectives of our sponsor in August of 2016. 

At the request of David Allsop from Boeing, we also connected a few people from our NAVAIR visits to 
discuss the issue of deriving MDAO parametrics from high-fidelity models, or more generally having 
some type of bi-directionality between parametric models and higher fidelity simulations (which can 
"break" the parametric chains). Dr. Dave McCormick who runs the MDAO lab for Northrop Grumman 
gave a relevant presentation at the April National Defense Industrial Association (NDIA) Modeling and 
Simulation bi-monthly committee meeting on some of challenges, which we believe are relevant to 
future research, such as: 

■ Rapid re-parameterization of completely new concepts 

■ Ability to incorporate static models 

■ Ability to bring in static changes "underneath" the parameterization 

■ Ability to incrementally add to parameterization 

■ Ability to rapidly alter the sizing logic behind models 
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13 Part II Summary 


This report has presented overviews of our SERC research to contribute to the NAVAIR Systems 
Engineering Transformation (SET). This report provides a high-level summary of the research-related 
events, results and deliverables. Our research from the early research tasks of RT-48/118/141 has 
transitioned from investigating the most holistic and advanced approaches to MCE and more generally 
Digital Engineering (DE) to demonstrate the art-of-the-possible in the context of the surrogate pilot 
experiments. The path forward to transitioning to DE presents challenges and opportunities, both 
technical and sociotechnical. The research has determined both industry and DoD are advancing pieces 
of the envisioned DE ecosystem and the technology is available to bring the DE ecosystem into 
existence, but there are still many research opportunities needed to fully realize the goals of a complete 
DE transformation. There are at least three key enablers: (1) Information Technology (IT) infrastructure, 
(2) workforce, and (3) policy to foster a new model for acquisition. 

IT infrastructure broadly includes all of the underlying computational technologies that have helped the 
Internet change our lives. These include a well-designed and secure suite of computational 
technologies, such as modeling and simulation, and semantic technology. The technologies could 
enable completeness and consistency checks, as well as methodological guidance for the creation of 
digital information from all types of media throughout multiple tools in the system lifecycle. The 
formalization of this information as models, provides future opportunities to leverage technologies that 
allow for semantically-rich data exchange, semantic reconciliation, and reasoning about information 
that would promote enhanced communication across stakeholders and provide the ability for computer 
augmented SE decision support. The modeling infrastructure for our surrogate pilot environment is a 
key step to enable an AST. We have an operational environment based on OpenMBEE, commercial tools 
and some integration and interoperability technologies. We look to the follow-on research task of RT- 
195 to connect a government-side of the AST with industry surrogate contractors. 

NAVAIR has made strides to advance the workforce through system modeling training. Our surrogate 
pilots are also providing demonstration and examples for mission and system modeling. The 
combination of basic training and examples created using best practices modeling methods, provides 
some initial steps toward understanding how different types of modeling artifacts can replace tradition 
document-based acquisition processes and artifacts. The surrogate pilot experiments to execute the SET 
Framework can provide a baseline for a next generation model-based acquisition paradigm. Acceptance 
of a DE ecosystem by the workforce can be another enabler to the DE transformation, and research 
suggests that education and training can make it successful. These demonstrations of the technology to 
the workforce should be enablers that reach more broadly throughout the DoD. Our interactions with 
other DoD stakeholders suggests that they too are ready to make the transformation in both policy and 
culture to enable the DE ecosystem. 
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14 Acronyms and Abbreviation 


This section provides a list of some of the terms used throughout the paper. The model lexicon should 


have all of these terms and 

many others. 

AADL 

Architecture Analysis & Design Language 

ACAT 

Acquisition Category 

AFT 

Architecture Framework Tool of NASA/JPL 

AGI 

Analytical Graphics, Inc. 

AGM 

Acquisition Guidance Model 

ANSI 

American National Standards Institute 

AP233 

Application Protocol 233 

ATL 

ATLAS Transformation Language 

ASR 

Alternative System Review 

AST 

Authoritative Source of Truth 

AVSI 

Aerospace Vehicle Systems Institute 

BDD 

SysML Block Definition Diagram 

BN 

Bayesian Network 

BNF 

Backus Naur Form 

BOM 

Bill of Material 

BPML 

Business Process Modeling Language 

CAD 

Computer-Aided Design 

CASE 

Computer-Aided Software Engineering 

CDR 

Critical Design Review 

CEO 

Chief Executive Officer 

CESUN 

International Engineering Systems Symposium 

CMM 

Capability Maturity Model 

CMMI 

Capability Maturity Model Integration 

CORBA 

Common Object Requesting Broker Architecture 

CREATE 

Computational Research and Engineering for Acquisition Tools and Environments 

CWM 

Common Warehouse Metamodel 

dB 

Decibel 

DBMS 

Database Management System 

DAG 

Defense Acquisition Guidebook 

DARPA 

Defense Advanced Research Project Agency 

DAU 

Defense Acquisition University 

DCDR 

Digital design from Critical Design Review (CDR) 

DE 

Digital Engineering 

DEWG 

Digital Engineering Working Group 

DL 

Descriptive Logic 

DoD 

Department of Defense 

DoDAF 

Department of Defense Architectural Framework 

DoE 

Design of Experiments 

DSL 

Domain Specific Languages 

DSM 

Domain Specific Modeling 

DSML 

Domain Specific Modeling Language 

E/DRAP 

Engineering Data Requirements Agreement Plan 

ERS 

Engineered Resilient Systems 

FAA 

Federal Aviation Administration 
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FMEA 

FMI 

FMU 

GAO 

GFI 

HPC 

HPCM 

HW 

l&l 

IBM 

IBD 

ICD 

ICTB 

IDEFO 

IEEE 

INCOSE 

IPR 

IRL 

ISEF 

ISO 

IT 

IWC 

JCIDS 

JEO 

JSF 

JPL 

KPP 

KSA 

Linux 

LOC 

M&S 

MARTE 

MATRIXx 

MBEE 

MBSE 

MBT 

MC/DC 

MCE 

MDA® 

MDAO 

MDD™ 

MDE 

MDSD 

MDSE 

MIC 

MMM 

MoDAF 

MOE 


Failure Modes and Effects Analysis 

Functional Mockup Interface 

Functional Mockup Unit 

Government Accounting Office 

Government Furnished Information 

High Performance Computing 

High Performance Computing Modernization 

Hardware 

Integration and Interoperability 
International Business Machines 
SysML Internal Block Diagram 
Interface Control Document 
Integrated Capability Technical Baseline 
learn DEFinition for Function Modeling 
Institute of Electrical and Electronics Engineers 
International Council on Systems Engineering 
Integration Problem Report 
Integration Readiness Level 

Integrated System Engineering Framework developed by Army's TARDEC 

International Organization for Standardization 

Information Technology 

Integrated Warfighter Capability 

Joint Capabilities Integration and Development System 

Jupiter Europa Orbiter project at NASA/JPL 

Joint Strike Fighter 

Jet Propulsion Laboratory of NASA 

Key Performance Parameter 

Key System Attributes 

An operating system created by Linus Torvalds 

Lines of Code 

Modeling and Simulation 

Modeling and Analysis of Real Time Embedded systems 

Product family for model-based control system design produced by National 

Instruments; Similar to Simulink 

Model-based Engineering Environment 

Model-based System Engineering 

Model Based Testing 

Modified Condition/Decision 

Model-centric engineering 

Model Driven Architecture® 

Multidisciplinary Design, Analysis and Optimization 

Model Driven Development 

Model Driven Engineering 

Model Driven Software Development 

Model Driven Software Engineering 

Model Integrated Computing 

Modeling Maturity Model 

United Kingdom Ministry of Defence Architectural Framework 
Measure of Effectiveness 
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MOF 

Meta Object Facility 

MOP 

Measure of Performance 

MVS 

Multiple Virtual Storage 

NASA 

National Aeronautics and Space Administration 

NAVAIR 

U.S. Navy Naval Air Systems Command 

NAVSEA 

U.S. Naval Sea Systems Command 

NDA 

Non-disclosure Agreement 

NDIA 

National Defense Industrial Association 

NEAR 

Naval Enterprise Architecture Repository 

NPS 

Naval Postgraduate School 

OCL 

Object Constraint Language 

ODASD(SE) 

Office of Deputy Assistant Secretary of Defense for Systems Engineering 

OMG 

Object Management Group 

00 

Object oriented 

OSD 

Office of the Secretary of Defense 

OSLC 

Open Services for Lifecycle Collaboration 

0V1 

Operational View 1 - type of DoDAF diagram 

OWL 

Web Ontology Language 

PDM 

Product Data Management 

PDR 

Preliminary Design Review 

PES 

Physical Exchange Specification 

PIA 

Proprietary Information Agreement 

PIM 

Platform Independent Model 

PLM 

Product Lifecycle Management 

POR 

Program of Record 

PRR 

Production Readiness Review 

PSM 

Platform Specific Model 

QMU 

Quantification of Margins and Uncertainty 

RT 

Research Task 

RFI 

Request for Information 

RFP 

Request for Proposal 

ROI 

Return On Investment 

SAVI 

System Architecture Virtual Integration 

SE 

System Engineering 

SERC 

Systems Engineering Research Center 

SETR 

System Engineering Technical Review 

Simulink/Stateflow 

Product family for model-based control system produced by The Mathworks 

SCR 

Software Cost Reduction 

SDD 

Software Design Document 

SE 

System Engineering 

SET 

Systems Engineering Transformation (as defined by NAVAIR) 

SFR 

System Functional Review 

SLOC 

Software Lines of Code 

SME 

Subject Matter Expert 

SOAP 

A protocol for exchanging XML-based messages - originally stood for Simple Object 
Access Protocol 

SoS 

System of Systems 

SOW 

Statement of Work 

SRR 

System Requirements Review 
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SRS 

Software Requirement Specification 

STOVL 

Short takeoff and vertical landing 

SVR 

System Verification Review 

SW 

Software 

SysML 

System Modeling Language 

TARDEC 

US Army Tank Automotive Research 

TBD 

To Be Determined 

TRL 

Technology Readiness Level 

TRR 

Test Readiness Review 

UML 

Unified Modeling Language 

XMI 

XML Metadata Interchange 

XML 

extensible Markup Language 

US 

United States 

XSLT 

extensible Stylesheet Language family (XSL) Transformation 

xUML 

Executable UML 

Unix 

An operating system with trademark held by the Open Group 

UQ 

Uncertainty Quantification 

VHDL 

Verilog Hardware Description Language 

V&V 

Verification and Validation 

VxWorks 

Operating system designed for embedded systems and owned by WindRiver 


15 Trademarks 


Analysis Server is a registered trademark of Phoenix Integration, Inc. 

Astah SysML is Copyright of Change Vision, Inc. 

BridgePoint is a registered trademark of Mentor Graphics. 

Cameo Simulation Toolkit is a registered trademark of No Magic, Inc. 

CORE is a registered trademark of Vitech Corporation. 

IBM™ is a trademark of the IBM Corporation 
iGrafx is a registered trademark of iGrafx, LCC. 

Java™ and J2EE™ are trademark of SUN Microsystems 
Java is trademarked by Sun Microsystems, Inc. 

LDRA is a registered trademark of Trademark of LDRA Ltd. and Subsidiaries. 

Linux is a registered trademark of Linux Mark Institute. 

Mathworks, Simulink, and Stateflow are registered trademarks of The Mathworks, Inc. 

MagicDraw is a trademark of No Magic, Inc. 

MATRIXx is a registered trademark of National Instruments. 

Microsoft®, Windows®, Windows NT®, Windows Server® and Windows VistaTM are either registered 
trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. 
ModelCenter, is a registered trademark of Phoenix Integration, Inc. 
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Modelica® is a registered trademark of the Modelica Association. 

Object Management Group (OMG): OMG's Registered Trademarks include: MDA®, Model Driven 
Architecture®, UML®, CORBA®, CORBA Academy®, XMI® 

OMG's Trademarks include, CWM™, Model Based Application Development™, MDD™, Model Based 
Development™, Model Based Management™, Model Based Programming™, Model Driven Application 
Development™, Model Driven Development™ 

Model Driven Programming™, Model Driven Systems™, OMG Interface Definition Language (IDL)™, 
Unified Modeling Language™, «UML»™ 

OMG®, MDA®, UML®, MOF®, XMI®, SysML™, BPML™ are registered trademarks or trademarks of the 
Object Management Group. 

Oracle and Java are registered trademarks of Oracle, Inc. and/or its affiliates. 

ParaMagic is a registered trademark of InterCAX, Inc. 

PHX ModelCenter is a registered trademark of Phoenix Integration, Inc. 

PowerPoint is a registered trademark of Microsoft, Inc. 

Real-time Studio Professional is a registered trademark of ARTiSAN Software Tools, Inc. 

Rhapsody is a registered trademark of Telelogic/IBM. 

Rose XDE is a registered trademark of IBM. 

SCADE is copyrighted to Esterel Technologies. 

Simulink is a registered trademark of The MathWorks. 

Stateflow is a registered trademark of The MathWorks. 

Statemate is a registered trademark of Telelogic/IBM. 

STK is a registered trademark of Analytical Graphics, Incorporated (AGI), Inc. 

UNIX is a registered trademark of The Open Group. 

VAPS is registered at eNGENUITY Technologies. 

VectorCAST is a registered trademark of Vector Software, Inc. 

Visio is a registered trademark of Microsoft, Inc. 

VxWorks is a registered trademark of Wind River Systems, Inc. 

Windows is a registered trademark of Microsoft Corporation in the United States and other countries. 
XML™ is a trademark of W3C 

All other trademarks belong to their respective organizations. 
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